2024-09-15T20:52:41.345 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T20:52:41.360 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:52:41.439 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657 branch: main description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905657' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: false bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITjjYUo1wr2qsMQzOfQQXtAAO9KGSOHbSGAl9+/L8CioDWJEY7NncwQCgI7U/J+UToKesvHvs8hXnhVGPEwSmQ= tasks: - install: null - ceph: create_rbd_pool: false log-ignorelist: - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(SLOW_OPS\) - slow request - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - ec_lost_unfound: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T20:52:41.439 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T20:52:41.440 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T20:52:41.440 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T20:52:41.441 INFO:teuthology.task.internal:Checking packages... 2024-09-15T20:52:41.463 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T20:52:41.463 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T20:52:41.463 INFO:teuthology.packaging:ref: None 2024-09-15T20:52:41.464 INFO:teuthology.packaging:tag: None 2024-09-15T20:52:41.464 INFO:teuthology.packaging:branch: main 2024-09-15T20:52:41.464 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:52:41.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-15T20:52:41.715 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930.gfc3e0bff 2024-09-15T20:52:41.718 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T20:52:41.724 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T20:52:41.724 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T20:52:41.742 INFO:teuthology.task.internal:Saving configuration 2024-09-15T20:52:41.758 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T20:52:41.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T20:52:41.790 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:47:43.732110', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITjjYUo1wr2qsMQzOfQQXtAAO9KGSOHbSGAl9+/L8CioDWJEY7NncwQCgI7U/J+UToKesvHvs8hXnhVGPEwSmQ='} 2024-09-15T20:52:41.790 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T20:52:41.796 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-15T20:52:41.797 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T20:52:41.830 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fed411b3ac0>, signals=[15]) 2024-09-15T20:52:41.830 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T20:52:41.836 INFO:teuthology.task.internal:Opening connections... 2024-09-15T20:52:41.836 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T20:52:41.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:52:41.914 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T20:52:41.920 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-15T20:52:42.003 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-15T20:52:42.003 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T20:52:42.062 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T20:52:42.063 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T20:52:42.064 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-15T20:52:42.083 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T20:52:42.090 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T20:52:42.096 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T20:52:42.096 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:52:42.118 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T20:52:42.124 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T20:52:42.124 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:52:42.477 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T20:52:42.507 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T20:52:42.521 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T20:52:42.521 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T20:52:42.521 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:52:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:52:42.539 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-15T20:52:42.597 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-15T20:52:42.597 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-15T20:52:42.597 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-15T20:52:45.539 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.2 MB 00:01 2024-09-15T20:52:47.623 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-15T20:52:53.291 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-15T20:52:55.649 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-15T20:52:56.704 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-15T20:53:02.201 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-15T20:53:04.165 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T20:53:04.166 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T20:53:04.252 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T20:53:04.259 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T20:53:04.260 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:53:04.261 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-15T20:53:04.261 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:53:04.261 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-15T20:53:04.261 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-15T20:53:04.261 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T20:53:04.611 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 320 kB/s | 23 kB 00:00 2024-09-15T20:53:05.036 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-15T20:53:05.811 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-15T20:53:06.070 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-15T20:53:06.070 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:53:06.070 INFO:teuthology.orchestra.run.smithi181.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-15T20:53:06.754 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T20:53:07.094 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T20:53:07.094 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T20:53:08.055 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T20:53:08.055 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T20:53:10.491 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T20:53:11.127 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:53:11.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:53:14.023 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:53:14.157 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:53:18.975 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:53:19.024 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:53:23.875 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:54:02.247 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:54:02.295 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:54:02.628 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:54:02.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T20:54:02.629 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T20:54:02.629 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T20:54:03.450 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T20:54:03.450 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:54:03.450 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T20:54:03.450 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.450 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.451 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.451 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.451 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:54:03.451 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T20:54:03.809 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:54:03.889 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.890 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:54:03.890 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T20:54:03.890 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:54:03.890 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:54:03.890 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T20:54:03.890 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:54:03.891 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T20:54:03.949 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T20:54:03.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T20:54:03.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-15T20:54:03.981 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T20:54:03.982 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T20:54:03.983 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T20:54:03.983 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-15T20:54:03.983 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:54:03.983 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T20:54:03.983 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T20:54:03.984 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-505.el9.x86_64 2024-09-15T20:54:03.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T20:54:04.641 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-15T20:54:05.989 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T20:54:06.025 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-15T20:54:06.028 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T20:54:06.044 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.k3pZDsaFbq 2024-09-15T20:54:06.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.k3pZDsaFbq 2024-09-15T20:54:06.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.k3pZDsaFbq 2024-09-15T20:54:06.255 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.k3pZDsaFbq is 6KB 2024-09-15T20:54:06.305 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.k3pZDsaFbq 2024-09-15T20:54:06.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T20:54:06.382 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T20:54:06.382 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T20:54:06.382 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T20:54:06.382 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T20:54:06.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T20:54:06.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-15T20:54:36.566 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T20:54:36.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T20:54:36.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:55:36.629 DEBUG:teuthology.orchestra.remote:timed out 2024-09-15T20:55:45.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T20:55:45.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:55:45.997 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-15T20:55:46.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T20:55:46.347 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T20:55:46.348 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-15T20:55:46.362 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-505.el9.x86_64 2024-09-15T20:55:46.363 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T20:55:46.363 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:55:46.363 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T20:55:47.364 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T20:55:47.364 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:55:47.443 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-15T20:55:47.466 DEBUG:teuthology.parallel:result is None 2024-09-15T20:55:47.466 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T20:55:47.474 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T20:55:47.474 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:55:47.490 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T20:55:47.527 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T20:55:47.535 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T20:55:47.535 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:55:47.564 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T20:55:47.571 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T20:55:47.571 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:55:47.631 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:55:47.642 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:55:47.644 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T20:55:47.653 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T20:55:47.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:55:47.707 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T20:55:47.717 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T20:55:47.717 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:55:47.759 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:55:47.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:55:47.884 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:55:47.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:55:48.000 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T20:55:48.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:55:48.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-15T20:55:48.150 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T20:55:48.336 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T20:55:48.344 INFO:teuthology.task.internal:Starting timer... 2024-09-15T20:55:48.344 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T20:55:48.354 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T20:55:48.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-15T20:55:48.399 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-15T20:55:48.401 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T20:55:48.401 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-15T20:55:48.462 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-15T20:55:48.462 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-15T20:55:48.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T20:55:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:55:48.527 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T20:55:48.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-15T20:55:48.595 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T20:55:48.604 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T20:55:48.607 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T20:55:48.714 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T20:55:48.726 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T20:55:48.727 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T20:59:32.455 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-15T20:59:32.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T20:59:32.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:59:32.535 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-15T20:59:32.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-15T20:59:32.620 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T20:59:32.630 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T20:59:32.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:59:32.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:59:32.687 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T20:59:32.701 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T20:59:32.725 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-15T20:59:32.736 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-15T20:59:32.750 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T20:59:32.764 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T20:59:32.820 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-15T20:59:32.823 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T20:59:32.823 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-15T20:59:32.823 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:59:32.823 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:59:32.824 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:59:32.824 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T20:59:32.825 INFO:teuthology.run_tasks:Running task install... 2024-09-15T20:59:32.835 DEBUG:teuthology.task.install:project ceph 2024-09-15T20:59:32.835 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T20:59:32.836 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T20:59:32.836 INFO:teuthology.task.install:Using flavor: default 2024-09-15T20:59:32.841 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T20:59:32.841 INFO:teuthology.task.install:extra packages: [] 2024-09-15T20:59:32.841 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'tag': None, 'wait_for_package': False} 2024-09-15T20:59:32.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:59:32.990 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/ 2024-09-15T20:59:32.990 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4930.gfc3e0bff 2024-09-15T20:59:33.100 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T20:59:33.100 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T20:59:33.100 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T20:59:33.126 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T20:59:33.127 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/fc3e0bff7468fd55478adbae7f75508b749124b6/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T20:59:33.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T20:59:33.268 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T20:59:33.332 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-15T20:59:33.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-15T20:59:33.578 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-15T20:59:33.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-15T20:59:34.154 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 323 kB/s | 84 kB 00:00 2024-09-15T20:59:34.471 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 88 kB/s | 24 kB 00:00 2024-09-15T20:59:34.772 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 7.3 kB/s | 1.9 kB 00:00 2024-09-15T20:59:36.746 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-15T20:59:40.806 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 20 MB 00:02 2024-09-15T20:59:45.240 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T20:59:46.641 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 119 kB/s | 19 kB 00:00 2024-09-15T20:59:47.761 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T20:59:53.201 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T20:59:55.576 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T20:59:55.579 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 11 M 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.5 M 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 22 M 2024-09-15T20:59:55.580 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 25 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 755 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 508 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 5.5 M 2024-09-15T20:59:55.581 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 45 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 143 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 320 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 305 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 99 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-15T20:59:55.582 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout:Install 31 Packages 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-15T20:59:55.583 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T20:59:55.584 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 62 M 2024-09-15T20:59:55.584 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T20:59:56.427 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-base-19.3.0-4930.gfc3e0bff.el9.x86 9.6 MB/s | 5.5 MB 00:00 2024-09-15T20:59:56.494 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-selinux-19.3.0-4930.gfc3e0bff.el9. 378 kB/s | 25 kB 00:00 2024-09-15T20:59:56.661 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): ceph-radosgw-19.3.0-4930.gfc3e0bff.el9. 13 MB/s | 11 MB 00:00 2024-09-15T20:59:56.702 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): libcephfs2-19.3.0-4930.gfc3e0bff.el9.x8 3.5 MB/s | 755 kB 00:00 2024-09-15T20:59:56.728 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): libradosstriper1-19.3.0-4930.gfc3e0bff. 7.5 MB/s | 508 kB 00:00 2024-09-15T20:59:56.778 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): python3-ceph-argparse-19.3.0-4930.gfc3e 903 kB/s | 45 kB 00:00 2024-09-15T20:59:56.853 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): python3-ceph-common-19.3.0-4930.gfc3e0b 1.9 MB/s | 143 kB 00:00 2024-09-15T20:59:56.920 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-cephfs-19.3.0-4930.gfc3e0bff.el 2.4 MB/s | 161 kB 00:00 2024-09-15T20:59:57.129 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): ceph-common-19.3.0-4930.gfc3e0bff.el9.x 17 MB/s | 22 MB 00:01 2024-09-15T20:59:57.221 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): librgw2-19.3.0-4930.gfc3e0bff.el9.x86_ 11 MB/s | 5.5 MB 00:00 2024-09-15T20:59:57.254 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-rados-19.3.0-4930.gfc3e0bff.el 958 kB/s | 320 kB 00:00 2024-09-15T20:59:57.280 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-rbd-19.3.0-4930.gfc3e0bff.el9. 2.0 MB/s | 305 kB 00:00 2024-09-15T20:59:57.305 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): python3-rgw-19.3.0-4930.gfc3e0bff.el9. 1.2 MB/s | 99 kB 00:00 2024-09-15T20:59:57.739 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 221 kB/s | 107 kB 00:00 2024-09-15T20:59:57.831 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 310 kB/s | 171 kB 00:00 2024-09-15T20:59:57.856 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 291 kB/s | 160 kB 00:00 2024-09-15T20:59:57.873 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 338 kB/s | 45 kB 00:00 2024-09-15T20:59:57.982 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 390 kB/s | 42 kB 00:00 2024-09-15T20:59:58.148 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-09-15T20:59:58.215 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-09-15T20:59:58.249 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 358 kB/s | 36 kB 00:00 2024-09-15T20:59:58.274 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 755 kB/s | 221 kB 00:00 2024-09-15T20:59:58.325 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-15T20:59:58.341 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 376 kB/s | 47 kB 00:00 2024-09-15T20:59:58.358 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 822 kB/s | 26 kB 00:00 2024-09-15T20:59:58.392 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-15T20:59:58.417 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-09-15T20:59:58.459 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-15T20:59:58.501 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 839 kB 00:00 2024-09-15T20:59:58.626 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-15T20:59:58.668 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-09-15T20:59:58.810 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): librados2-19.3.0-4930.gfc3e0bff.el9.x8 11 MB/s | 3.5 MB 00:00 2024-09-15T20:59:58.902 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): librbd1-19.3.0-4930.gfc3e0bff.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-15T20:59:58.907 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T20:59:58.907 INFO:teuthology.orchestra.run.smithi181.stdout:Total 19 MB/s | 62 MB 00:03 2024-09-15T20:59:59.018 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T20:59:59.055 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T20:59:59.055 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T20:59:59.721 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T20:59:59.721 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:01.227 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:01.303 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-15T21:00:01.332 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-15T21:00:01.365 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-15T21:00:01.397 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T21:00:01.430 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-15T21:00:01.630 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T21:00:01.701 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-15T21:00:02.051 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-15T21:00:02.159 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T21:00:02.252 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T21:00:02.341 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 8/35 2024-09-15T21:00:02.361 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T21:00:02.423 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 9/35 2024-09-15T21:00:02.469 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-15T21:00:02.530 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 11/35 2024-09-15T21:00:02.600 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T21:00:02.630 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T21:00:02.694 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 13/35 2024-09-15T21:00:02.729 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-15T21:00:02.837 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-15T21:00:02.871 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-15T21:00:02.920 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-15T21:00:02.953 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-15T21:00:02.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-15T21:00:03.015 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T21:00:03.056 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-15T21:00:03.400 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-15T21:00:03.445 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T21:00:03.545 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 23/35 2024-09-15T21:00:03.575 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 24/35 2024-09-15T21:00:03.611 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-15T21:00:04.103 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-15T21:00:04.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T21:00:04.227 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 27/35 2024-09-15T21:00:04.298 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 28/35 2024-09-15T21:00:04.333 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 29/35 2024-09-15T21:00:06.674 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T21:00:06.713 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T21:00:07.303 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 30/35 2024-09-15T21:00:07.350 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T21:00:07.445 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 31/35 2024-09-15T21:00:07.445 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T21:00:07.445 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T21:00:07.445 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:07.484 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T21:00:17.703 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-15T21:00:17.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:18.711 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/35 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T21:00:18.737 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:18.739 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T21:00:18.822 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-15T21:00:18.822 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_6 6/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 7/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9. 8/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x8 9/35 2024-09-15T21:00:20.045 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 10/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 11/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 12/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 13/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-15T21:00:20.046 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-15T21:00:20.047 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-15T21:00:20.048 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-15T21:00:20.048 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 32/35 2024-09-15T21:00:20.048 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-15T21:00:20.048 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/35 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.986 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T21:00:20.987 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.988 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:20.989 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:21.325 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-15T21:00:21.895 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:28 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:22.027 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 46 M 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 161 k 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T21:00:22.028 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 46 M 2024-09-15T21:00:22.029 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 201 M 2024-09-15T21:00:22.030 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:00:22.286 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): libcephsqlite-19.3.0-4930.gfc3e0bff.el9. 774 kB/s | 161 kB 00:00 2024-09-15T21:00:22.319 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-15T21:00:22.444 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 402 kB/s | 64 kB 00:00 2024-09-15T21:00:23.595 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): ceph-test-19.3.0-4930.gfc3e0bff.el9.x86_ 30 MB/s | 46 MB 00:01 2024-09-15T21:00:23.597 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:00:23.597 INFO:teuthology.orchestra.run.smithi181.stdout:Total 29 MB/s | 46 MB 00:01 2024-09-15T21:00:23.625 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:00:23.645 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:00:23.645 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:00:24.014 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:00:24.014 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:24.528 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:24.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T21:00:24.652 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T21:00:24.680 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T21:00:29.031 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/4 2024-09-15T21:00:29.120 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T21:00:31.492 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/4 2024-09-15T21:00:31.493 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/4 2024-09-15T21:00:31.493 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/4 2024-09-15T21:00:31.493 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:31.810 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T21:00:31.811 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T21:00:31.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:31.811 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:32.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-15T21:00:32.736 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:39 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:32.882 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 6.6 k 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:00:32.885 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 2.4 M 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 1.5 M 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 291 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 4.8 M 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 17 M 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T21:00:32.886 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T21:00:32.887 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:32.888 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout:Install 37 Packages 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 M 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 109 M 2024-09-15T21:00:32.889 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:00:33.379 INFO:teuthology.orchestra.run.smithi181.stdout:(1/37): ceph-19.3.0-4930.gfc3e0bff.el9.x86_64.r 47 kB/s | 6.6 kB 00:00 2024-09-15T21:00:33.538 INFO:teuthology.orchestra.run.smithi181.stdout:(2/37): ceph-mgr-19.3.0-4930.gfc3e0bff.el9.x86_ 5.1 MB/s | 1.5 MB 00:00 2024-09-15T21:00:33.613 INFO:teuthology.orchestra.run.smithi181.stdout:(3/37): ceph-mds-19.3.0-4930.gfc3e0bff.el9.x86_ 6.4 MB/s | 2.4 MB 00:00 2024-09-15T21:00:33.680 INFO:teuthology.orchestra.run.smithi181.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4930.gfc3e 4.3 MB/s | 291 kB 00:00 2024-09-15T21:00:33.805 INFO:teuthology.orchestra.run.smithi181.stdout:(5/37): ceph-mon-19.3.0-4930.gfc3e0bff.el9.x86_ 11 MB/s | 4.8 MB 00:00 2024-09-15T21:00:34.122 INFO:teuthology.orchestra.run.smithi181.stdout:(6/37): ceph-osd-19.3.0-4930.gfc3e0bff.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-15T21:00:34.417 INFO:teuthology.orchestra.run.smithi181.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 343 kB/s | 253 kB 00:00 2024-09-15T21:00:34.534 INFO:teuthology.orchestra.run.smithi181.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-15T21:00:34.584 INFO:teuthology.orchestra.run.smithi181.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 231 kB/s | 106 kB 00:00 2024-09-15T21:00:34.718 INFO:teuthology.orchestra.run.smithi181.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 679 kB/s | 125 kB 00:00 2024-09-15T21:00:34.785 INFO:teuthology.orchestra.run.smithi181.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-15T21:00:34.852 INFO:teuthology.orchestra.run.smithi181.stdout:(12/37): python3-cryptography-36.0.1-4.el9.x86_ 1.2 MB/s | 1.2 MB 00:01 2024-09-15T21:00:34.886 INFO:teuthology.orchestra.run.smithi181.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-09-15T21:00:34.936 INFO:teuthology.orchestra.run.smithi181.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-15T21:00:34.961 INFO:teuthology.orchestra.run.smithi181.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-15T21:00:34.986 INFO:teuthology.orchestra.run.smithi181.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-15T21:00:35.012 INFO:teuthology.orchestra.run.smithi181.stdout:(17/37): python3-mako-1.1.4-6.el9.noarch.rpm 761 kB/s | 172 kB 00:00 2024-09-15T21:00:35.046 INFO:teuthology.orchestra.run.smithi181.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 495 kB/s | 42 kB 00:00 2024-09-15T21:00:35.079 INFO:teuthology.orchestra.run.smithi181.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-15T21:00:35.104 INFO:teuthology.orchestra.run.smithi181.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-15T21:00:35.130 INFO:teuthology.orchestra.run.smithi181.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 530 kB/s | 43 kB 00:00 2024-09-15T21:00:35.155 INFO:teuthology.orchestra.run.smithi181.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-15T21:00:35.180 INFO:teuthology.orchestra.run.smithi181.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 716 kB/s | 18 kB 00:00 2024-09-15T21:00:35.205 INFO:teuthology.orchestra.run.smithi181.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 927 kB/s | 23 kB 00:00 2024-09-15T21:00:35.231 INFO:teuthology.orchestra.run.smithi181.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-15T21:00:35.256 INFO:teuthology.orchestra.run.smithi181.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-15T21:00:35.290 INFO:teuthology.orchestra.run.smithi181.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-09-15T21:00:35.315 INFO:teuthology.orchestra.run.smithi181.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 908 kB/s | 46 kB 00:00 2024-09-15T21:00:35.340 INFO:teuthology.orchestra.run.smithi181.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T21:00:35.365 INFO:teuthology.orchestra.run.smithi181.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-09-15T21:00:35.391 INFO:teuthology.orchestra.run.smithi181.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-15T21:00:35.417 INFO:teuthology.orchestra.run.smithi181.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-15T21:00:35.441 INFO:teuthology.orchestra.run.smithi181.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-15T21:00:35.467 INFO:teuthology.orchestra.run.smithi181.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-15T21:00:35.492 INFO:teuthology.orchestra.run.smithi181.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-15T21:00:35.517 INFO:teuthology.orchestra.run.smithi181.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-09-15T21:00:35.617 INFO:teuthology.orchestra.run.smithi181.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 35 kB/s | 22 kB 00:00 2024-09-15T21:00:35.626 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:00:35.626 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-15T21:00:35.777 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:00:35.817 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:00:35.817 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:00:36.334 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:00:36.335 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:37.737 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:37.812 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-15T21:00:37.861 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-15T21:00:37.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-15T21:00:37.969 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-15T21:00:38.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-15T21:00:38.071 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-15T21:00:38.163 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-15T21:00:38.249 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-15T21:00:38.290 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-15T21:00:38.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-15T21:00:38.356 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-15T21:00:38.390 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-15T21:00:38.423 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-15T21:00:38.464 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-15T21:00:38.501 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-15T21:00:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-15T21:00:38.591 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-15T21:00:38.632 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-15T21:00:38.713 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-15T21:00:38.859 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-15T21:00:38.939 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-15T21:00:38.992 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-15T21:00:39.031 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-15T21:00:39.080 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-15T21:00:39.152 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-15T21:00:39.337 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-15T21:00:39.386 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-15T21:00:39.463 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-15T21:00:39.552 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-15T21:00:39.601 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-15T21:00:39.727 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-15T21:00:39.903 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 32/37 2024-09-15T21:00:39.937 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T21:00:39.964 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 33/37 2024-09-15T21:00:39.964 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:39.964 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T21:00:39.964 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:00:39.965 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T21:00:39.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:41.421 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 34/37 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T21:00:41.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:41.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T21:00:41.921 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 35/37 2024-09-15T21:00:41.921 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:41.921 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T21:00:41.922 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:00:41.922 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T21:00:41.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:42.147 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T21:00:42.171 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 36/37 2024-09-15T21:00:42.171 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:42.172 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T21:00:42.172 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:00:42.172 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T21:00:42.172 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:42.236 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 37/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 3/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 4/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 5/37 2024-09-15T21:00:44.002 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9. 6/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-15T21:00:44.003 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-15T21:00:44.004 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-15T21:00:44.004 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-15T21:00:44.004 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-15T21:00:44.004 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-15T21:00:44.004 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-15T21:00:44.005 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-15T21:00:44.008 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-15T21:00:44.009 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-15T21:00:44.009 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-15T21:00:44.009 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-15T21:00:44.014 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-15T21:00:44.014 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-15T21:00:45.035 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-15T21:00:45.035 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:45.035 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:00:45.035 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T21:00:45.036 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T21:00:45.037 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:45.038 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:45.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-15T21:00:46.010 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:46.099 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:00:46.141 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:46.142 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:00:46.142 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:46.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-15T21:00:46.771 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:53 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:46.902 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 784 k 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 784 k 2024-09-15T21:00:46.903 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 791 k 2024-09-15T21:00:46.904 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:00:47.191 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.3.0-4930.gfc3e0bff.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-15T21:00:47.192 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:00:47.192 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-15T21:00:47.192 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:00:47.197 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:00:47.197 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:00:47.203 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:00:47.203 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:47.275 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:47.711 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:00:47.802 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:00:48.173 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:48.437 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:48.665 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T21:00:49.243 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:49.382 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:49.383 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:49.383 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T21:00:49.383 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 145 k 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 145 k 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 432 k 2024-09-15T21:00:49.384 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:00:49.580 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.3.0-4930.gfc3e0b 742 kB/s | 145 kB 00:00 2024-09-15T21:00:49.580 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:00:49.580 INFO:teuthology.orchestra.run.smithi181.stdout:Total 736 kB/s | 145 kB 00:00 2024-09-15T21:00:49.581 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:00:49.588 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:00:49.588 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:00:49.644 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:00:49.645 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:49.784 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:49.847 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T21:00:49.871 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T21:00:49.872 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:00:49.872 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T21:00:49.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff. 1/1 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:50.786 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:50.990 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-15T21:00:51.569 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:51.665 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:00:51.708 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:51.709 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:00:51.709 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:00:51.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T21:00:52.338 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:59 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:00:52.471 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:00:52.472 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 14 M 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 34 k 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 16 k 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-15T21:00:52.473 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-15T21:00:52.474 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:00:52.475 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 19 M 2024-09-15T21:00:52.475 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 159 M 2024-09-15T21:00:52.475 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:00:53.191 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): ceph-prometheus-alerts-19.3.0-4930.gfc3 101 kB/s | 16 kB 00:00 2024-09-15T21:00:53.216 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): ceph-grafana-dashboards-19.3.0-4930.gfc 182 kB/s | 34 kB 00:00 2024-09-15T21:00:53.400 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 269 kB 00:00 2024-09-15T21:00:53.550 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-15T21:00:53.693 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): ceph-mgr-dashboard-19.3.0-4930.gfc3e0bf 21 MB/s | 14 MB 00:00 2024-09-15T21:00:53.751 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.4 MB/s | 863 kB 00:00 2024-09-15T21:00:53.784 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 229 kB/s | 19 kB 00:00 2024-09-15T21:00:53.818 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.0 MB/s | 548 kB 00:00 2024-09-15T21:00:53.844 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-15T21:00:53.869 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.7 MB/s | 144 kB 00:00 2024-09-15T21:00:53.894 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T21:00:53.969 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.4 MB/s | 2.0 MB 00:00 2024-09-15T21:00:53.973 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:00:53.973 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 19 MB 00:01 2024-09-15T21:00:54.023 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:00:54.041 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:00:54.041 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:00:54.314 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:00:54.314 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:00:54.907 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:00:54.978 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-15T21:00:55.022 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-15T21:00:55.068 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-15T21:00:55.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-15T21:00:55.191 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-15T21:00:55.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-15T21:00:55.420 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-15T21:00:55.479 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-15T21:00:55.507 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-15T21:00:55.540 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 10/12 2024-09-15T21:00:58.751 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 11/12 2024-09-15T21:00:58.802 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T21:01:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 12/12 2024-09-15T21:01:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el 1/12 2024-09-15T21:01:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noa 2/12 2024-09-15T21:01:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9 3/12 2024-09-15T21:01:00.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-15T21:01:00.875 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-15T21:01:01.193 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-15T21:01:01.193 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:01.193 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T21:01:01.194 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:01.195 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:01.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T21:01:02.009 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:09 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:02.145 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 7.4 M 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:01:02.146 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-15T21:01:02.147 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-15T21:01:02.148 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:02.148 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-15T21:01:02.148 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-15T21:01:02.148 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:02.634 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-15T21:01:02.701 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-15T21:01:02.743 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-09-15T21:01:02.793 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-15T21:01:02.844 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 771 kB/s | 38 kB 00:00 2024-09-15T21:01:02.986 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-49 15 MB/s | 7.4 MB 00:00 2024-09-15T21:01:03.169 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.4 MB/s | 3.0 MB 00:00 2024-09-15T21:01:03.303 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-09-15T21:01:03.336 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-devel-3.9.19-8.el9.x86_64.rpm 699 kB/s | 245 kB 00:00 2024-09-15T21:01:03.370 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 3.7 MB/s | 250 kB 00:00 2024-09-15T21:01:03.470 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-15T21:01:04.954 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-15T21:01:04.955 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:04.955 INFO:teuthology.orchestra.run.smithi181.stdout:Total 14 MB/s | 38 MB 00:02 2024-09-15T21:01:05.306 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:05.331 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:05.331 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:05.765 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:05.765 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:06.558 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:06.631 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T21:01:06.668 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T21:01:06.767 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T21:01:06.812 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T21:01:06.997 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T21:01:07.019 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T21:01:07.104 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T21:01:07.755 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T21:01:07.874 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T21:01:09.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T21:01:10.202 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T21:01:10.236 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T21:01:12.908 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 12/12 2024-09-15T21:01:12.908 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0 1/12 2024-09-15T21:01:12.908 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T21:01:12.908 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T21:01:12.909 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T21:01:12.910 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T21:01:13.377 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:13.378 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:13.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-15T21:01:14.328 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:14.465 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:14.469 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:14.469 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:01:14.469 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:14.469 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 49 k 2024-09-15T21:01:14.470 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:01:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T21:01:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T21:01:14.470 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T21:01:14.471 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T21:01:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T21:01:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T21:01:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T21:01:14.472 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:14.472 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-15T21:01:14.473 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:14.686 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 258 kB/s | 26 kB 00:00 2024-09-15T21:01:14.711 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 154 kB/s | 19 kB 00:00 2024-09-15T21:01:14.795 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-15T21:01:14.820 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): ceph-mgr-rook-19.3.0-4930.gfc3e0bff.el9 211 kB/s | 49 kB 00:00 2024-09-15T21:01:14.845 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-09-15T21:01:14.879 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-09-15T21:01:14.912 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-15T21:01:14.938 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 247 kB/s | 14 kB 00:00 2024-09-15T21:01:14.963 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 375 kB/s | 54 kB 00:00 2024-09-15T21:01:14.996 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-15T21:01:15.022 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.7 MB/s | 90 kB 00:00 2024-09-15T21:01:15.047 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 703 kB/s | 59 kB 00:00 2024-09-15T21:01:15.081 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-15T21:01:15.083 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:15.083 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.6 MB/s | 2.2 MB 00:00 2024-09-15T21:01:15.146 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:15.160 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:15.161 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:15.250 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:15.251 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:15.761 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:15.936 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T21:01:15.990 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T21:01:16.055 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T21:01:16.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T21:01:16.132 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T21:01:16.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T21:01:16.344 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T21:01:16.401 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T21:01:17.118 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T21:01:17.165 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T21:01:17.198 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T21:01:17.247 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T21:01:17.268 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T21:01:18.116 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 13/13 2024-09-15T21:01:18.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/13 2024-09-15T21:01:18.116 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T21:01:18.117 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T21:01:18.118 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T21:01:18.118 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T21:01:18.448 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:18.449 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:18.678 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T21:01:19.253 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:19.389 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 154 k 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:01:19.390 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-15T21:01:19.391 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:19.392 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-15T21:01:19.392 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-15T21:01:19.392 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:19.791 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4930.gfc3e0bff.e 722 kB/s | 154 kB 00:00 2024-09-15T21:01:19.875 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-15T21:01:19.909 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.7 MB/s | 58 kB 00:00 2024-09-15T21:01:19.933 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-15T21:01:20.217 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 388 kB/s | 248 kB 00:00 2024-09-15T21:01:24.409 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.2 MB/s | 6.0 MB 00:04 2024-09-15T21:01:24.411 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:24.411 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.4 MB/s | 7.0 MB 00:05 2024-09-15T21:01:24.485 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:24.497 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:24.497 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:24.597 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:24.597 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:24.896 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:25.067 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T21:01:25.119 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T21:01:25.623 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T21:01:25.703 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T21:01:25.780 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T21:01:25.806 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T21:01:26.555 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 6/6 2024-09-15T21:01:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/6 2024-09-15T21:01:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T21:01:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T21:01:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T21:01:26.556 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T21:01:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T21:01:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:26.829 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:26.829 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:26.830 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:27.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-15T21:01:27.677 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:27.811 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:27.811 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 900 k 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 980 k 2024-09-15T21:01:27.812 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 2.9 M 2024-09-15T21:01:27.813 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:28.069 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 519 kB/s | 80 kB 00:00 2024-09-15T21:01:28.218 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.3.0-4930.gfc3e0bff.el9.x86_ 2.9 MB/s | 900 kB 00:00 2024-09-15T21:01:28.219 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:28.219 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.3 MB/s | 980 kB 00:00 2024-09-15T21:01:28.231 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:28.253 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:28.253 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:28.333 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:28.333 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:28.527 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:28.663 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T21:01:28.734 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T21:01:29.432 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2/2 2024-09-15T21:01:29.433 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/2 2024-09-15T21:01:29.723 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T21:01:29.723 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:29.723 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:29.724 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T21:01:29.724 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:29.724 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:30.047 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-15T21:01:30.637 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:30.776 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-4930.gfc3e0bff.el9 ceph-noarch 285 k 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:30.777 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:30.778 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:30.778 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:30.778 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 285 k 2024-09-15T21:01:30.778 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-15T21:01:30.778 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:31.007 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.3.0-4930.gfc3e0bff.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-15T21:01:31.008 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:31.008 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-15T21:01:31.008 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:31.017 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:31.017 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:31.048 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:31.048 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:31.267 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:31.335 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:01:31.351 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:01:31.351 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:01:31.351 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T21:01:31.351 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:32.388 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 1/1 2024-09-15T21:01:32.388 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:32.388 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:32.388 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-4930.gfc3e0bff.el9.noarch 2024-09-15T21:01:32.388 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:32.389 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:32.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-15T21:01:33.242 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:33.380 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:33.380 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 127 k 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 127 k 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 456 k 2024-09-15T21:01:33.381 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:33.588 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.3.0-4930.gfc3e0bff.el9.x86_64 614 kB/s | 127 kB 00:00 2024-09-15T21:01:33.588 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:33.588 INFO:teuthology.orchestra.run.smithi181.stdout:Total 609 kB/s | 127 kB 00:00 2024-09-15T21:01:33.589 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:33.594 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:33.594 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:33.657 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:33.658 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:33.806 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:33.910 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:34.346 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:34.628 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:34.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-15T21:01:35.493 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:42 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:35.585 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:35.632 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:35.633 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:35.633 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:35.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-15T21:01:36.276 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:36.410 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 32 k 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-09-15T21:01:36.411 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-09-15T21:01:36.412 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:36.566 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.3.0-4930.gfc3e0bff.el9.x86_6 205 kB/s | 32 kB 00:00 2024-09-15T21:01:36.567 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:36.567 INFO:teuthology.orchestra.run.smithi181.stdout:Total 203 kB/s | 32 kB 00:00 2024-09-15T21:01:36.567 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:36.571 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:36.571 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:36.592 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:36.592 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:36.678 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:36.805 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:37.196 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:37.435 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:37.622 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-15T21:01:38.184 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:38.274 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:38.317 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:38.318 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:38.318 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:38.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-15T21:01:38.953 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:45 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:39.048 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:39.094 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:39.095 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:39.095 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:39.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-15T21:01:39.755 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:39.848 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:39.892 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:39.893 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:39.963 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-15T21:01:40.528 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:47 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:40.619 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:40.662 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:40.663 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:40.663 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:40.728 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-15T21:01:41.296 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:41.387 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:41.431 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:41.432 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:41.432 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:41.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-15T21:01:42.064 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:42.155 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 is already installed. 2024-09-15T21:01:42.200 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:42.200 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-15T21:01:42.201 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:42.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-15T21:01:42.831 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:49 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:42.965 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 85 k 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:42.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:42.967 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 85 k 2024-09-15T21:01:42.967 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 231 k 2024-09-15T21:01:42.967 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:43.172 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 421 kB/s | 85 kB 00:00 2024-09-15T21:01:43.172 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:43.172 INFO:teuthology.orchestra.run.smithi181.stdout:Total 417 kB/s | 85 kB 00:00 2024-09-15T21:01:43.172 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:43.178 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:43.178 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:43.241 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:43.241 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:43.386 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:43.484 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:43.896 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:44.168 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:44.169 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:44.169 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:44.169 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:01:44.169 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:44.169 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:44.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-15T21:01:45.017 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:52 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:45.154 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 3.2 M 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:45.155 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.2 M 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 14 M 2024-09-15T21:01:45.156 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:45.490 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 9.5 MB/s | 3.2 MB 00:00 2024-09-15T21:01:45.491 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:45.491 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.5 MB/s | 3.2 MB 00:00 2024-09-15T21:01:45.491 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:45.500 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:45.501 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:45.583 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:45.583 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:46.029 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:46.216 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:46.242 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:46.242 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T21:01:46.242 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T21:01:46.242 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:01:46.243 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T21:01:46.243 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:47.219 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:47.220 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:47.220 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:47.220 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:01:47.220 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:47.220 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:47.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-15T21:01:48.035 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:55 ago on Sun 15 Sep 2024 08:59:53 PM UTC. 2024-09-15T21:01:48.174 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-4930.gfc3e0bff.el9 ceph 170 k 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-15T21:01:48.175 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-15T21:01:48.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:48.176 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 170 k 2024-09-15T21:01:48.176 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 490 k 2024-09-15T21:01:48.176 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-15T21:01:48.402 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.3.0-4930.gfc3e0bff.el9.x86_64.rpm 754 kB/s | 170 kB 00:00 2024-09-15T21:01:48.403 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-15T21:01:48.403 INFO:teuthology.orchestra.run.smithi181.stdout:Total 748 kB/s | 170 kB 00:00 2024-09-15T21:01:48.403 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-15T21:01:48.410 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-15T21:01:48.410 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-15T21:01:48.474 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-15T21:01:48.474 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-15T21:01:48.634 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-15T21:01:48.733 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:49.184 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:49.475 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 1/1 2024-09-15T21:01:49.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:49.476 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-15T21:01:49.476 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-4930.gfc3e0bff.el9.x86_64 2024-09-15T21:01:49.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:49.476 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-15T21:01:49.740 DEBUG:teuthology.parallel:result is None 2024-09-15T21:01:49.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T21:01:49.889 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T21:01:49.911 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-4930.gfc3e0bff.el9 2024-09-15T21:01:49.911 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930.gfc3e0bff.el9 2024-09-15T21:01:49.912 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930.gfc3e0bff is installed. 2024-09-15T21:01:49.914 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T21:01:49.914 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:49.914 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T21:01:49.975 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T21:01:49.976 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:49.976 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T21:01:50.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T21:01:50.107 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T21:01:50.107 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:50.107 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T21:01:50.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T21:01:50.238 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T21:01:50.238 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:50.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T21:01:50.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T21:01:50.370 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T21:01:50.468 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T21:01:50.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-15T21:01:50.494 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T21:01:50.494 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T21:01:50.560 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T21:01:50.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T21:01:50.637 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T21:01:50.637 INFO:tasks.ceph:config {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-15T21:01:50.637 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657', 'branch': 'main', 'description': 'rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905657', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITjjYUo1wr2qsMQzOfQQXtAAO9KGSOHbSGAl9+/L8CioDWJEY7NncwQCgI7U/J+UToKesvHvs8hXnhVGPEwSmQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'create_rbd_pool': False, 'log-ignorelist': ['objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(SLOW_OPS\\)', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'cluster': 'ceph'}}, {'ec_lost_unfound': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T21:01:50.638 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T21:01:50.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T21:01:50.747 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:50.748 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T21:01:50.804 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T21:01:50.804 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-15T21:01:50.859 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T21:01:50.859 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:01:50.859 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 21:01:46.500517486 +0000 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 20:59:13.911960155 +0000 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 20:59:13.911960155 +0000 2024-09-15T21:01:50.860 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 20:59:13.911960155 +0000 2024-09-15T21:01:50.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T21:01:50.923 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T21:01:50.923 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T21:01:50.923 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.00022487 s, 2.3 MB/s 2024-09-15T21:01:50.924 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T21:01:50.981 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 21:01:46.501517466 +0000 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 20:59:14.268953149 +0000 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 20:59:14.268953149 +0000 2024-09-15T21:01:51.038 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 20:59:14.268953149 +0000 2024-09-15T21:01:51.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T21:01:51.101 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T21:01:51.101 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T21:01:51.101 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000202161 s, 2.5 MB/s 2024-09-15T21:01:51.102 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T21:01:51.159 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-15T21:01:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T21:01:51.214 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 21:01:46.501517466 +0000 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 20:59:14.582946988 +0000 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 20:59:14.582946988 +0000 2024-09-15T21:01:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 20:59:14.582946988 +0000 2024-09-15T21:01:51.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T21:01:51.278 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T21:01:51.278 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T21:01:51.278 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000195361 s, 2.6 MB/s 2024-09-15T21:01:51.279 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T21:01:51.336 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-15 21:01:46.501517466 +0000 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-15 20:59:14.906940630 +0000 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-15 20:59:14.906940630 +0000 2024-09-15T21:01:51.392 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-15 20:59:14.906940630 +0000 2024-09-15T21:01:51.393 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T21:01:51.455 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-15T21:01:51.455 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-15T21:01:51.455 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000227034 s, 2.3 MB/s 2024-09-15T21:01:51.456 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T21:01:51.512 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T21:01:51.513 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T21:01:51.513 INFO:tasks.ceph:Generating config... 2024-09-15T21:01:51.514 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-09-15T21:01:51.514 INFO:tasks.ceph:[global] ms type = async 2024-09-15T21:01:51.514 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T21:01:51.514 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T21:01:51.514 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T21:01:51.515 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T21:01:51.515 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T21:01:51.515 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore write v2 = False 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T21:01:51.515 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T21:01:51.516 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T21:01:51.517 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T21:01:51.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T21:01:51.609 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T21:01:51.611 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T21:01:51.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T21:01:51.718 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.181'), ('mon.b', '[v2:172.21.15.181:3301,v1:172.21.15.181:6790]'), ('mon.c', '[v2:172.21.15.181:3302,v1:172.21.15.181:6791]')] 2024-09-15T21:01:51.719 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790],[v2:172.21.15.181:3302,v1:172.21.15.181:6791]', 'mon election default strategy': 3, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': False, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-15T21:01:51.719 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:51.719 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T21:01:51.774 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.181 --addv b '[v2:172.21.15.181:3301,v1:172.21.15.181:6790]' --addv c '[v2:172.21.15.181:3302,v1:172.21.15.181:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:01:51.860 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:01:51.860 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: generated fsid c31fa425-5a82-4117-b11e-63b4944cd2c3 2024-09-15T21:01:51.860 INFO:teuthology.orchestra.run.smithi181.stdout:setting min_mon_release = quincy 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 0 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:fsid c31fa425-5a82-4117-b11e-63b4944cd2c3 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:last_changed 2024-09-15T21:01:51.858531+0000 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-09-15T21:01:51.858531+0000 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:min_mon_release 17 (quincy) 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:election_strategy: 3 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:0: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.a 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:1: [v2:172.21.15.181:3301/0,v1:172.21.15.181:6790/0] mon.b 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:2: [v2:172.21.15.181:3302/0,v1:172.21.15.181:6791/0] mon.c 2024-09-15T21:01:51.861 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T21:01:51.862 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T21:01:51.877 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c31fa425-5a82-4117-b11e-63b4944cd2c3... 2024-09-15T21:01:51.878 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T21:01:51.960 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-15T21:01:51.960 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-15T21:01:51.960 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = cephx 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.961 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-15T21:01:51.962 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.963 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = "172.21.15.181,[v2:172.21.15.181:3301,v1:172.21.15.181:6790],[v2:172.21.15.181:3302,v1:172.21.15.181:6791]" 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: mon election default strategy = 3 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: ms type = async 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = c31fa425-5a82-4117-b11e-63b4944cd2c3 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-15T21:01:51.964 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.965 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression algorithm = lz4 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression mode = aggressive 2024-09-15T21:01:51.966 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore write v2 = False 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore zero block detection = True 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 20 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify cached snaps = True 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify missing on start = True 2024-09-15T21:01:51.967 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock override recovery settings = True 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue = debug_random 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: osd op queue cut off = debug_random 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-15T21:01:51.968 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 20 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.969 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: mon scrub interval = 300 2024-09-15T21:01:51.970 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.b] 2024-09-15T21:01:51.971 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.c] 2024-09-15T21:01:51.972 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T21:01:51.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T21:01:52.067 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T21:01:52.067 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:52.067 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T21:01:52.082 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:52.083 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T21:01:52.138 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:01:52.138 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:52.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T21:01:52.138 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T21:01:52.212 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:01:52.212 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:01:52.268 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T21:01:52.268 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T21:01:52.268 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T21:01:52.360 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T21:01:52.362 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T21:01:52.363 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T21:01:52.363 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T21:01:52.363 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-15T21:01:52.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T21:01:52.387 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T21:01:52.387 INFO:tasks.ceph:role: osd.0 2024-09-15T21:01:52.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:01:52.388 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T21:01:52.460 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T21:01:52.954 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T21:01:52.956 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T21:01:52.956 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T21:01:53.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T21:01:53.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T21:01:53.189 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T21:01:53.189 INFO:tasks.ceph:role: osd.1 2024-09-15T21:01:53.189 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:01:53.190 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T21:01:53.262 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T21:01:53.262 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T21:01:53.262 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T21:01:53.262 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T21:01:53.263 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T21:01:53.720 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T21:01:53.722 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T21:01:53.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T21:01:53.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T21:01:53.838 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T21:01:53.903 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T21:01:53.903 INFO:tasks.ceph:role: osd.2 2024-09-15T21:01:53.903 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:01:53.903 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T21:01:53.975 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T21:01:53.976 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T21:01:54.478 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T21:01:54.480 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T21:01:54.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T21:01:54.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T21:01:54.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T21:01:54.724 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-15T21:01:54.725 INFO:tasks.ceph:role: osd.3 2024-09-15T21:01:54.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:01:54.725 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-15T21:01:54.797 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T21:01:54.797 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T21:01:54.797 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T21:01:54.797 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-15T21:01:54.797 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T21:01:54.798 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-15T21:01:54.798 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T21:01:54.798 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-15T21:01:54.798 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T21:01:54.798 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T21:01:55.262 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-15T21:01:55.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-15T21:01:55.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-15T21:01:55.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T21:01:55.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:01:55.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:55.556+0000 7f010ece8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T21:01:55.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:55.556+0000 7f010ece8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T21:01:55.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:55.556+0000 7f010ece8640 -1 bdev(0x557735cb7800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-15T21:01:55.559 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:55.556+0000 7f010ece8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T21:01:58.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T21:01:58.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:01:58.600 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:58.597+0000 7f3e985db640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T21:01:58.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:58.597+0000 7f3e985db640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T21:01:58.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:58.597+0000 7f3e985db640 -1 bdev(0x558f16787800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-15T21:01:58.601 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:01:58.597+0000 7f3e985db640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T21:02:01.508 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T21:02:01.538 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:02:01.628 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:01.624+0000 7fc8f2a64640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T21:02:01.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:01.625+0000 7fc8f2a64640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T21:02:01.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:01.625+0000 7fc8f2a64640 -1 bdev(0x55bd4ff01800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-15T21:02:01.629 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:01.625+0000 7fc8f2a64640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T21:02:04.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T21:02:04.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:02:04.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:04.889+0000 7f161af1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T21:02:04.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:04.890+0000 7f161af1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T21:02:04.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:04.890+0000 7f161af1e640 -1 bdev(0x564d3c5ad800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-15T21:02:04.893 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-15T21:02:04.890+0000 7f161af1e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T21:02:07.800 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T21:02:07.830 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T21:02:07.830 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:07.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T21:02:07.899 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:07.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T21:02:07.965 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:07.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T21:02:08.031 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:08.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T21:02:08.097 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:08.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T21:02:08.165 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T21:02:08.165 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBAS+dmWRtYFRAAh1rqB08IZSboqne/8xAAUA== 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBDS+dmX+gyIRAAWDyitWNrwYIB6J4zkc/ItQ== 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBGS+dmgwyqIxAA/lRK6AagLMmJS6ytYJ6mqg== 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-15T21:02:08.228 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBJS+dmf/NLJRAAqUtiFgsTg+43v3+9Acw8Bg== 2024-09-15T21:02:08.229 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-15T21:02:08.229 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQBMS+dmkyIXNRAAmxsDSALyXPu6K5R1v5zI8g== 2024-09-15T21:02:08.229 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T21:02:08.320 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T21:02:08.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T21:02:08.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T21:02:08.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T21:02:08.523 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T21:02:08.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T21:02:08.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T21:02:08.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T21:02:08.670 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T21:02:08.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T21:02:08.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T21:02:08.855 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T21:02:08.920 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T21:02:09.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T21:02:09.041 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T21:02:09.096 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T21:02:09.096 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T21:02:09.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T21:02:09.138 INFO:tasks.ceph.mon.a:Started 2024-09-15T21:02:09.138 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T21:02:09.139 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T21:02:09.141 INFO:tasks.ceph.mon.b:Started 2024-09-15T21:02:09.141 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T21:02:09.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T21:02:09.143 INFO:tasks.ceph.mon.c:Started 2024-09-15T21:02:09.143 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph config set mgr mgr_pool false --force 2024-09-15T21:02:15.338 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T21:02:15.339 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T21:02:15.339 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T21:02:15.341 INFO:tasks.ceph.mgr.x:Started 2024-09-15T21:02:15.342 DEBUG:tasks.ceph:set 0 configs 2024-09-15T21:02:15.342 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T21:02:15.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T21:02:15.517 INFO:teuthology.orchestra.run.smithi181.stderr:adjusted tunables profile to default 2024-09-15T21:02:15.525 INFO:tasks.ceph:check_enable_crimson: False 2024-09-15T21:02:15.525 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T21:02:15.525 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:15.525 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T21:02:15.549 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:15.550 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T21:02:15.614 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:15.615 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T21:02:15.679 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-15T21:02:15.680 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T21:02:15.745 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 92896e8a-721b-45df-b061-1807b46a1f81 0 2024-09-15T21:02:15.970 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-09-15T21:02:15.981 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 84736d86-cc39-4c9a-9154-48a566b2eb39 1 2024-09-15T21:02:16.163 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-15T21:02:16.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 198fc7a7-b64a-4498-95bc-18995cfaa86a 2 2024-09-15T21:02:16.435 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-09-15T21:02:16.447 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new f42cf73c-6ff4-4301-b660-9e56cc9bed0c 3 2024-09-15T21:02:16.625 INFO:teuthology.orchestra.run.smithi181.stdout:3 2024-09-15T21:02:16.637 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T21:02:16.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T21:02:16.639 INFO:tasks.ceph.osd.0:Started 2024-09-15T21:02:16.640 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T21:02:16.640 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T21:02:16.642 INFO:tasks.ceph.osd.1:Started 2024-09-15T21:02:16.642 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T21:02:16.642 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T21:02:16.644 INFO:tasks.ceph.osd.2:Started 2024-09-15T21:02:16.644 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T21:02:16.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T21:02:16.646 INFO:tasks.ceph.osd.3:Started 2024-09-15T21:02:16.647 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:16.895 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:16.895 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":6,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:16.620369+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:16.906 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T21:02:16.906 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T21:02:17.308 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:17.468 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-15T21:02:17.469 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":6,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:16.620369+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:17.479 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-15T21:02:18.093 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-09-15T21:02:18.089+0000 7ff4e96e4640 -1 Falling back to public interface 2024-09-15T21:02:18.103 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:02:18.099+0000 7fef30011640 -1 Falling back to public interface 2024-09-15T21:02:18.120 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:02:18.116+0000 7f164060c640 -1 Falling back to public interface 2024-09-15T21:02:18.337 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:02:18.334+0000 7f5562517640 -1 Falling back to public interface 2024-09-15T21:02:20.904 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T21:02:20.901+0000 7fd1ececf640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T21:02:20.904 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-15T21:02:20.901+0000 7fd1ececf640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T21:02:22.137 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-09-15T21:02:22.134+0000 7ff4e96e4640 -1 osd.2 0 log_to_monitors true 2024-09-15T21:02:22.186 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:02:22.183+0000 7fef30011640 -1 osd.1 0 log_to_monitors true 2024-09-15T21:02:22.459 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:02:22.456+0000 7f164060c640 -1 osd.0 0 log_to_monitors true 2024-09-15T21:02:22.646 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:02:22.643+0000 7f5562517640 -1 osd.3 0 log_to_monitors true 2024-09-15T21:02:23.883 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:23.925 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:02:23.922+0000 7f163be86640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T21:02:23.926 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:02:23.922+0000 7f555e4a6640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T21:02:23.941 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:02:23.938+0000 7f555a2cd640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:02:23.941 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:02:23.938+0000 7f1637cad640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:02:24.372 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-15T21:02:24.372 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":8,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:23.906233+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:24.388 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-15T21:02:24.675 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-09-15T21:02:24.671+0000 7ff4e5673640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T21:02:24.694 INFO:tasks.ceph.osd.2.smithi181.stderr:2024-09-15T21:02:24.691+0000 7ff4dd492640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:02:24.705 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:02:24.702+0000 7fef2b91e640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T21:02:24.721 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:02:24.718+0000 7fef2373d640 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:02:30.791 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:31.079 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-15T21:02:31.079 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":10,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:25.929224+0000","last_up_change":"2024-09-15T21:02:24.917559+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6817","nonce":132522244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6819","nonce":132522244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6823","nonce":132522244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6821","nonce":132522244}]},"public_addr":"172.21.15.181:6817/132522244","cluster_addr":"172.21.15.181:6819/132522244","heartbeat_back_addr":"172.21.15.181:6823/132522244","heartbeat_front_addr":"172.21.15.181:6821/132522244","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6825","nonce":2479280537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6827","nonce":2479280537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6831","nonce":2479280537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6829","nonce":2479280537}]},"public_addr":"172.21.15.181:6825/2479280537","cluster_addr":"172.21.15.181:6827/2479280537","heartbeat_back_addr":"172.21.15.181:6831/2479280537","heartbeat_front_addr":"172.21.15.181:6829/2479280537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:31.090 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-15T21:02:31.090 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T21:02:31.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-15T21:02:31.091 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T21:02:31.374 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:31.387 INFO:teuthology.orchestra.run.smithi181.stdout:[{"version":4,"timestamp":"2024-09-15T21:02:24.703527+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"7018.555648"}]}] 2024-09-15T21:02:31.387 INFO:tasks.ceph_manager:config epoch is 4 2024-09-15T21:02:31.387 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T21:02:31.387 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T21:02:31.388 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T21:02:31.720 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:31.731 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":2893493449},{"type":"v1","addr":"172.21.15.181:6833","nonce":2893493449}]},"active_addr":"172.21.15.181:6833/2893493449","active_change":"2024-09-15T21:02:19.890542+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":3930214503}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":361621267}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2762645766}]}]} 2024-09-15T21:02:31.735 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T21:02:31.735 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T21:02:31.735 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:32.022 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:32.023 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":10,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:25.929224+0000","last_up_change":"2024-09-15T21:02:24.917559+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6817","nonce":132522244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6819","nonce":132522244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6823","nonce":132522244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6821","nonce":132522244}]},"public_addr":"172.21.15.181:6817/132522244","cluster_addr":"172.21.15.181:6819/132522244","heartbeat_back_addr":"172.21.15.181:6823/132522244","heartbeat_front_addr":"172.21.15.181:6821/132522244","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6825","nonce":2479280537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6827","nonce":2479280537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6831","nonce":2479280537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6829","nonce":2479280537}]},"public_addr":"172.21.15.181:6825/2479280537","cluster_addr":"172.21.15.181:6827/2479280537","heartbeat_back_addr":"172.21.15.181:6831/2479280537","heartbeat_front_addr":"172.21.15.181:6829/2479280537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:32.035 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T21:02:32.035 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:32.319 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:32.319 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":10,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:25.929224+0000","last_up_change":"2024-09-15T21:02:24.917559+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6817","nonce":132522244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6819","nonce":132522244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6823","nonce":132522244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6821","nonce":132522244}]},"public_addr":"172.21.15.181:6817/132522244","cluster_addr":"172.21.15.181:6819/132522244","heartbeat_back_addr":"172.21.15.181:6823/132522244","heartbeat_front_addr":"172.21.15.181:6821/132522244","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6825","nonce":2479280537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6827","nonce":2479280537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6831","nonce":2479280537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6829","nonce":2479280537}]},"public_addr":"172.21.15.181:6825/2479280537","cluster_addr":"172.21.15.181:6827/2479280537","heartbeat_back_addr":"172.21.15.181:6831/2479280537","heartbeat_front_addr":"172.21.15.181:6829/2479280537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:32.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T21:02:32.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T21:02:32.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T21:02:32.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T21:02:32.478 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:32.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T21:02:32.484 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:32.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:02:32.491 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:32.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:02:32.498 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:32.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:02:32.819 INFO:teuthology.orchestra.run.smithi181.stdout:38654705667 2024-09-15T21:02:32.832 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.1 2024-09-15T21:02:32.846 INFO:teuthology.orchestra.run.smithi181.stdout:38654705667 2024-09-15T21:02:32.860 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.0 2024-09-15T21:02:32.873 INFO:teuthology.orchestra.run.smithi181.stdout:38654705667 2024-09-15T21:02:32.887 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.3 2024-09-15T21:02:32.899 INFO:teuthology.orchestra.run.smithi181.stdout:38654705667 2024-09-15T21:02:32.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705667 for osd.2 2024-09-15T21:02:33.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T21:02:33.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:02:33.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:02:33.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:02:34.191 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:34.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-09-15T21:02:34.209 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:34.218 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:34.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-15T21:02:34.236 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:34.242 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:34.255 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2024-09-15T21:02:34.255 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:34.276 INFO:teuthology.orchestra.run.smithi181.stdout:38654705668 2024-09-15T21:02:34.290 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-09-15T21:02:34.290 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:34.290 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T21:02:34.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:34.610 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:34.610 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:34.622 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-15T21:02:33.897959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":81300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-15T21:02:34.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:34.905 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:34.906 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:34.918 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-15T21:02:33.897959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":81300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-15T21:02:34.919 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T21:02:34.919 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T21:02:34.919 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T21:02:34.919 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T21:02:35.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:35.236 INFO:teuthology.orchestra.run.smithi181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T21:02:35.249 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T21:02:35.249 INFO:teuthology.run_tasks:Running task ec_lost_unfound... 2024-09-15T21:02:35.275 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:02:35.559 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:35.559 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":10,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:02:25.929224+0000","last_up_change":"2024-09-15T21:02:24.917559+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6817","nonce":132522244}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6819","nonce":132522244}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6823","nonce":132522244}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":132522244},{"type":"v1","addr":"172.21.15.181:6821","nonce":132522244}]},"public_addr":"172.21.15.181:6817/132522244","cluster_addr":"172.21.15.181:6819/132522244","heartbeat_back_addr":"172.21.15.181:6823/132522244","heartbeat_front_addr":"172.21.15.181:6821/132522244","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6825","nonce":2479280537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6827","nonce":2479280537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6831","nonce":2479280537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6829","nonce":2479280537}]},"public_addr":"172.21.15.181:6825/2479280537","cluster_addr":"172.21.15.181:6827/2479280537","heartbeat_back_addr":"172.21.15.181:6831/2479280537","heartbeat_front_addr":"172.21.15.181:6829/2479280537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:02:35.572 INFO:tasks.ec_lost_unfound.ceph_manager:[] 2024-09-15T21:02:35.572 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T21:02:35.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:35.852 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:35.853 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:35.865 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-15T21:02:33.897959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":81300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-15T21:02:35.866 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:36.145 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:36.145 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:36.158 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-15T21:02:35.898228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":108472,"kb_used_data":352,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378888,"statfs":{"total":386547056640,"available":386435981312,"internally_reserved":0,"allocated":360448,"data_stored":81300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27120,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344720,"statfs":{"total":96636764160,"available":96608993280,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27116,"kb_used_data":88,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":90112,"data_stored":20325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-09-15T21:02:36.158 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T21:02:36.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set lost_unfound k=2 m=2 crush-failure-domain=osd 2024-09-15T21:02:36.979 INFO:tasks.ec_lost_unfound.ceph_manager:creating pool_name unique_pool_0 2024-09-15T21:02:36.979 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure lost_unfound 2024-09-15T21:02:38.976 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'unique_pool_0' created 2024-09-15T21:02:38.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2024-09-15T21:02:39.973 INFO:teuthology.orchestra.run.smithi181.stderr:set pool 1 min_size to 2 2024-09-15T21:02:39.994 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-15T21:02:40.978 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-15T21:02:42.009 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2024-09-15T21:02:42.010 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2024-09-15T21:02:42.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T21:02:42.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T21:02:42.209 INFO:teuthology.orchestra.run.smithi181.stdout:38654705671 2024-09-15T21:02:42.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T21:02:42.211 INFO:teuthology.orchestra.run.smithi181.stdout:38654705671 2024-09-15T21:02:42.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:02:42.539 INFO:teuthology.orchestra.run.smithi181.stdout:38654705670 2024-09-15T21:02:42.540 INFO:teuthology.orchestra.run.smithi181.stdout:38654705670 2024-09-15T21:02:42.554 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-15T21:02:42.555 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2024-09-15T21:02:43.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T21:02:43.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:02:43.865 INFO:teuthology.orchestra.run.smithi181.stdout:38654705670 2024-09-15T21:02:43.869 INFO:teuthology.orchestra.run.smithi181.stdout:38654705670 2024-09-15T21:02:43.878 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2024-09-15T21:02:43.882 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705670 for osd.1 2024-09-15T21:02:44.879 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:02:44.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T21:02:45.180 INFO:teuthology.orchestra.run.smithi181.stdout:38654705671 2024-09-15T21:02:45.182 INFO:teuthology.orchestra.run.smithi181.stdout:38654705671 2024-09-15T21:02:45.196 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705671 for osd.1 2024-09-15T21:02:45.196 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:45.197 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705671 got 38654705671 for osd.0 2024-09-15T21:02:45.197 DEBUG:teuthology.parallel:result is None 2024-09-15T21:02:45.197 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-15T21:02:45.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:45.480 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:45.480 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:45.492 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-15T21:02:43.899483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108636,"kb_used_data":500,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378724,"statfs":{"total":386547056640,"available":386435813376,"internally_reserved":0,"allocated":512000,"data_stored":154776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974801+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974801+0000","last_undegraded":"2024-09-15T21:02:39.974801+0000","last_fullsized":"2024-09-15T21:02:39.974801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003329+0000","last_change":"2024-09-15T21:02:41.005604+0000","last_active":"2024-09-15T21:02:42.003329+0000","last_peered":"2024-09-15T21:02:42.003329+0000","last_clean":"2024-09-15T21:02:42.003329+0000","last_became_active":"2024-09-15T21:02:41.004622+0000","last_became_peered":"2024-09-15T21:02:41.004622+0000","last_unstale":"2024-09-15T21:02:42.003329+0000","last_undegraded":"2024-09-15T21:02:42.003329+0000","last_fullsized":"2024-09-15T21:02:42.003329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:46.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.973554+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.973554+0000","last_undegraded":"2024-09-15T21:02:39.973554+0000","last_fullsized":"2024-09-15T21:02:39.973554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003288+0000","last_change":"2024-09-15T21:02:41.002709+0000","last_active":"2024-09-15T21:02:42.003288+0000","last_peered":"2024-09-15T21:02:42.003288+0000","last_clean":"2024-09-15T21:02:42.003288+0000","last_became_active":"2024-09-15T21:02:41.002136+0000","last_became_peered":"2024-09-15T21:02:41.002136+0000","last_unstale":"2024-09-15T21:02:42.003288+0000","last_undegraded":"2024-09-15T21:02:42.003288+0000","last_fullsized":"2024-09-15T21:02:42.003288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:17:36.601796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974687+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974687+0000","last_undegraded":"2024-09-15T21:02:39.974687+0000","last_fullsized":"2024-09-15T21:02:39.974687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.975599+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.975599+0000","last_undegraded":"2024-09-15T21:02:39.975599+0000","last_fullsized":"2024-09-15T21:02:39.975599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.070628+0000","last_change":"2024-09-15T21:02:40.993255+0000","last_active":"2024-09-15T21:02:42.070628+0000","last_peered":"2024-09-15T21:02:42.070628+0000","last_clean":"2024-09-15T21:02:42.070628+0000","last_became_active":"2024-09-15T21:02:40.992835+0000","last_became_peered":"2024-09-15T21:02:40.992835+0000","last_unstale":"2024-09-15T21:02:42.070628+0000","last_undegraded":"2024-09-15T21:02:42.070628+0000","last_fullsized":"2024-09-15T21:02:42.070628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:59.355039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003649+0000","last_change":"2024-09-15T21:02:41.005652+0000","last_active":"2024-09-15T21:02:42.003649+0000","last_peered":"2024-09-15T21:02:42.003649+0000","last_clean":"2024-09-15T21:02:42.003649+0000","last_became_active":"2024-09-15T21:02:41.004773+0000","last_became_peered":"2024-09-15T21:02:41.004773+0000","last_unstale":"2024-09-15T21:02:42.003649+0000","last_undegraded":"2024-09-15T21:02:42.003649+0000","last_fullsized":"2024-09-15T21:02:42.003649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:50.374186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974795+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974795+0000","last_undegraded":"2024-09-15T21:02:39.974795+0000","last_fullsized":"2024-09-15T21:02:39.974795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974878+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974878+0000","last_undegraded":"2024-09-15T21:02:39.974878+0000","last_fullsized":"2024-09-15T21:02:39.974878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.976506+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.976506+0000","last_undegraded":"2024-09-15T21:02:39.976506+0000","last_fullsized":"2024-09-15T21:02:39.976506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003525+0000","last_change":"2024-09-15T21:02:41.002468+0000","last_active":"2024-09-15T21:02:42.003525+0000","last_peered":"2024-09-15T21:02:42.003525+0000","last_clean":"2024-09-15T21:02:42.003525+0000","last_became_active":"2024-09-15T21:02:41.002081+0000","last_became_peered":"2024-09-15T21:02:41.002081+0000","last_unstale":"2024-09-15T21:02:42.003525+0000","last_undegraded":"2024-09-15T21:02:42.003525+0000","last_fullsized":"2024-09-15T21:02:42.003525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:56.145668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003578+0000","last_change":"2024-09-15T21:02:41.005290+0000","last_active":"2024-09-15T21:02:42.003578+0000","last_peered":"2024-09-15T21:02:42.003578+0000","last_clean":"2024-09-15T21:02:42.003578+0000","last_became_active":"2024-09-15T21:02:41.004822+0000","last_became_peered":"2024-09-15T21:02:41.004822+0000","last_unstale":"2024-09-15T21:02:42.003578+0000","last_undegraded":"2024-09-15T21:02:42.003578+0000","last_fullsized":"2024-09-15T21:02:42.003578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:43.163756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003107+0000","last_change":"2024-09-15T21:02:41.002426+0000","last_active":"2024-09-15T21:02:42.003107+0000","last_peered":"2024-09-15T21:02:42.003107+0000","last_clean":"2024-09-15T21:02:42.003107+0000","last_became_active":"2024-09-15T21:02:41.001595+0000","last_became_peered":"2024-09-15T21:02:41.001595+0000","last_unstale":"2024-09-15T21:02:42.003107+0000","last_undegraded":"2024-09-15T21:02:42.003107+0000","last_fullsized":"2024-09-15T21:02:42.003107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:45:31.676166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003636+0000","last_change":"2024-09-15T21:02:41.006060+0000","last_active":"2024-09-15T21:02:42.003636+0000","last_peered":"2024-09-15T21:02:42.003636+0000","last_clean":"2024-09-15T21:02:42.003636+0000","last_became_active":"2024-09-15T21:02:41.005224+0000","last_became_peered":"2024-09-15T21:02:41.005224+0000","last_unstale":"2024-09-15T21:02:42.003636+0000","last_undegraded":"2024-09-15T21:02:42.003636+0000","last_fullsized":"2024-09-15T21:02:42.003636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:50:31.724159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003276+0000","last_change":"2024-09-15T21:02:41.002141+0000","last_active":"2024-09-15T21:02:42.003276+0000","last_peered":"2024-09-15T21:02:42.003276+0000","last_clean":"2024-09-15T21:02:42.003276+0000","last_became_active":"2024-09-15T21:02:41.001768+0000","last_became_peered":"2024-09-15T21:02:41.001768+0000","last_unstale":"2024-09-15T21:02:42.003276+0000","last_undegraded":"2024-09-15T21:02:42.003276+0000","last_fullsized":"2024-09-15T21:02:42.003276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:53.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":35680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":114688,"data_stored":31723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":39663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:02:45.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:45.778 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:45.779 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:45.791 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-15T21:02:43.899483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108636,"kb_used_data":500,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378724,"statfs":{"total":386547056640,"available":386435813376,"internally_reserved":0,"allocated":512000,"data_stored":154776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974801+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974801+0000","last_undegraded":"2024-09-15T21:02:39.974801+0000","last_fullsized":"2024-09-15T21:02:39.974801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003329+0000","last_change":"2024-09-15T21:02:41.005604+0000","last_active":"2024-09-15T21:02:42.003329+0000","last_peered":"2024-09-15T21:02:42.003329+0000","last_clean":"2024-09-15T21:02:42.003329+0000","last_became_active":"2024-09-15T21:02:41.004622+0000","last_became_peered":"2024-09-15T21:02:41.004622+0000","last_unstale":"2024-09-15T21:02:42.003329+0000","last_undegraded":"2024-09-15T21:02:42.003329+0000","last_fullsized":"2024-09-15T21:02:42.003329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:46.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.973554+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.973554+0000","last_undegraded":"2024-09-15T21:02:39.973554+0000","last_fullsized":"2024-09-15T21:02:39.973554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003288+0000","last_change":"2024-09-15T21:02:41.002709+0000","last_active":"2024-09-15T21:02:42.003288+0000","last_peered":"2024-09-15T21:02:42.003288+0000","last_clean":"2024-09-15T21:02:42.003288+0000","last_became_active":"2024-09-15T21:02:41.002136+0000","last_became_peered":"2024-09-15T21:02:41.002136+0000","last_unstale":"2024-09-15T21:02:42.003288+0000","last_undegraded":"2024-09-15T21:02:42.003288+0000","last_fullsized":"2024-09-15T21:02:42.003288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:17:36.601796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974687+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974687+0000","last_undegraded":"2024-09-15T21:02:39.974687+0000","last_fullsized":"2024-09-15T21:02:39.974687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.975599+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.975599+0000","last_undegraded":"2024-09-15T21:02:39.975599+0000","last_fullsized":"2024-09-15T21:02:39.975599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.070628+0000","last_change":"2024-09-15T21:02:40.993255+0000","last_active":"2024-09-15T21:02:42.070628+0000","last_peered":"2024-09-15T21:02:42.070628+0000","last_clean":"2024-09-15T21:02:42.070628+0000","last_became_active":"2024-09-15T21:02:40.992835+0000","last_became_peered":"2024-09-15T21:02:40.992835+0000","last_unstale":"2024-09-15T21:02:42.070628+0000","last_undegraded":"2024-09-15T21:02:42.070628+0000","last_fullsized":"2024-09-15T21:02:42.070628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:59.355039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003649+0000","last_change":"2024-09-15T21:02:41.005652+0000","last_active":"2024-09-15T21:02:42.003649+0000","last_peered":"2024-09-15T21:02:42.003649+0000","last_clean":"2024-09-15T21:02:42.003649+0000","last_became_active":"2024-09-15T21:02:41.004773+0000","last_became_peered":"2024-09-15T21:02:41.004773+0000","last_unstale":"2024-09-15T21:02:42.003649+0000","last_undegraded":"2024-09-15T21:02:42.003649+0000","last_fullsized":"2024-09-15T21:02:42.003649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:50.374186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974795+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974795+0000","last_undegraded":"2024-09-15T21:02:39.974795+0000","last_fullsized":"2024-09-15T21:02:39.974795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974878+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974878+0000","last_undegraded":"2024-09-15T21:02:39.974878+0000","last_fullsized":"2024-09-15T21:02:39.974878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.976506+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.976506+0000","last_undegraded":"2024-09-15T21:02:39.976506+0000","last_fullsized":"2024-09-15T21:02:39.976506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003525+0000","last_change":"2024-09-15T21:02:41.002468+0000","last_active":"2024-09-15T21:02:42.003525+0000","last_peered":"2024-09-15T21:02:42.003525+0000","last_clean":"2024-09-15T21:02:42.003525+0000","last_became_active":"2024-09-15T21:02:41.002081+0000","last_became_peered":"2024-09-15T21:02:41.002081+0000","last_unstale":"2024-09-15T21:02:42.003525+0000","last_undegraded":"2024-09-15T21:02:42.003525+0000","last_fullsized":"2024-09-15T21:02:42.003525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:56.145668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003578+0000","last_change":"2024-09-15T21:02:41.005290+0000","last_active":"2024-09-15T21:02:42.003578+0000","last_peered":"2024-09-15T21:02:42.003578+0000","last_clean":"2024-09-15T21:02:42.003578+0000","last_became_active":"2024-09-15T21:02:41.004822+0000","last_became_peered":"2024-09-15T21:02:41.004822+0000","last_unstale":"2024-09-15T21:02:42.003578+0000","last_undegraded":"2024-09-15T21:02:42.003578+0000","last_fullsized":"2024-09-15T21:02:42.003578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:43.163756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003107+0000","last_change":"2024-09-15T21:02:41.002426+0000","last_active":"2024-09-15T21:02:42.003107+0000","last_peered":"2024-09-15T21:02:42.003107+0000","last_clean":"2024-09-15T21:02:42.003107+0000","last_became_active":"2024-09-15T21:02:41.001595+0000","last_became_peered":"2024-09-15T21:02:41.001595+0000","last_unstale":"2024-09-15T21:02:42.003107+0000","last_undegraded":"2024-09-15T21:02:42.003107+0000","last_fullsized":"2024-09-15T21:02:42.003107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:45:31.676166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003636+0000","last_change":"2024-09-15T21:02:41.006060+0000","last_active":"2024-09-15T21:02:42.003636+0000","last_peered":"2024-09-15T21:02:42.003636+0000","last_clean":"2024-09-15T21:02:42.003636+0000","last_became_active":"2024-09-15T21:02:41.005224+0000","last_became_peered":"2024-09-15T21:02:41.005224+0000","last_unstale":"2024-09-15T21:02:42.003636+0000","last_undegraded":"2024-09-15T21:02:42.003636+0000","last_fullsized":"2024-09-15T21:02:42.003636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:50:31.724159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003276+0000","last_change":"2024-09-15T21:02:41.002141+0000","last_active":"2024-09-15T21:02:42.003276+0000","last_peered":"2024-09-15T21:02:42.003276+0000","last_clean":"2024-09-15T21:02:42.003276+0000","last_became_active":"2024-09-15T21:02:41.001768+0000","last_became_peered":"2024-09-15T21:02:41.001768+0000","last_unstale":"2024-09-15T21:02:42.003276+0000","last_undegraded":"2024-09-15T21:02:42.003276+0000","last_fullsized":"2024-09-15T21:02:42.003276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:53.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":35680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":114688,"data_stored":31723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":39663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:02:45.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:46.076 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:46.076 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:46.088 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-15T21:02:45.899767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108636,"kb_used_data":500,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378724,"statfs":{"total":386547056640,"available":386435813376,"internally_reserved":0,"allocated":512000,"data_stored":154776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974801+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974801+0000","last_undegraded":"2024-09-15T21:02:39.974801+0000","last_fullsized":"2024-09-15T21:02:39.974801+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003329+0000","last_change":"2024-09-15T21:02:41.005604+0000","last_active":"2024-09-15T21:02:42.003329+0000","last_peered":"2024-09-15T21:02:42.003329+0000","last_clean":"2024-09-15T21:02:42.003329+0000","last_became_active":"2024-09-15T21:02:41.004622+0000","last_became_peered":"2024-09-15T21:02:41.004622+0000","last_unstale":"2024-09-15T21:02:42.003329+0000","last_undegraded":"2024-09-15T21:02:42.003329+0000","last_fullsized":"2024-09-15T21:02:42.003329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:46.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.973554+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.973554+0000","last_undegraded":"2024-09-15T21:02:39.973554+0000","last_fullsized":"2024-09-15T21:02:39.973554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003288+0000","last_change":"2024-09-15T21:02:41.002709+0000","last_active":"2024-09-15T21:02:42.003288+0000","last_peered":"2024-09-15T21:02:42.003288+0000","last_clean":"2024-09-15T21:02:42.003288+0000","last_became_active":"2024-09-15T21:02:41.002136+0000","last_became_peered":"2024-09-15T21:02:41.002136+0000","last_unstale":"2024-09-15T21:02:42.003288+0000","last_undegraded":"2024-09-15T21:02:42.003288+0000","last_fullsized":"2024-09-15T21:02:42.003288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:17:36.601796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974687+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974687+0000","last_undegraded":"2024-09-15T21:02:39.974687+0000","last_fullsized":"2024-09-15T21:02:39.974687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.975599+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.975599+0000","last_undegraded":"2024-09-15T21:02:39.975599+0000","last_fullsized":"2024-09-15T21:02:39.975599+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.070628+0000","last_change":"2024-09-15T21:02:40.993255+0000","last_active":"2024-09-15T21:02:42.070628+0000","last_peered":"2024-09-15T21:02:42.070628+0000","last_clean":"2024-09-15T21:02:42.070628+0000","last_became_active":"2024-09-15T21:02:40.992835+0000","last_became_peered":"2024-09-15T21:02:40.992835+0000","last_unstale":"2024-09-15T21:02:42.070628+0000","last_undegraded":"2024-09-15T21:02:42.070628+0000","last_fullsized":"2024-09-15T21:02:42.070628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:59.355039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003649+0000","last_change":"2024-09-15T21:02:41.005652+0000","last_active":"2024-09-15T21:02:42.003649+0000","last_peered":"2024-09-15T21:02:42.003649+0000","last_clean":"2024-09-15T21:02:42.003649+0000","last_became_active":"2024-09-15T21:02:41.004773+0000","last_became_peered":"2024-09-15T21:02:41.004773+0000","last_unstale":"2024-09-15T21:02:42.003649+0000","last_undegraded":"2024-09-15T21:02:42.003649+0000","last_fullsized":"2024-09-15T21:02:42.003649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:50.374186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974795+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974795+0000","last_undegraded":"2024-09-15T21:02:39.974795+0000","last_fullsized":"2024-09-15T21:02:39.974795+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.974878+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.974878+0000","last_undegraded":"2024-09-15T21:02:39.974878+0000","last_fullsized":"2024-09-15T21:02:39.974878+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":2,"reported_epoch":14,"state":"creating+peering","last_fresh":"2024-09-15T21:02:39.976506+0000","last_change":"2024-09-15T21:02:38.968547+0000","last_active":"2024-09-15T21:02:38.968547+0000","last_peered":"2024-09-15T21:02:38.968547+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-15T21:02:39.976506+0000","last_undegraded":"2024-09-15T21:02:39.976506+0000","last_fullsized":"2024-09-15T21:02:39.976506+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003525+0000","last_change":"2024-09-15T21:02:41.002468+0000","last_active":"2024-09-15T21:02:42.003525+0000","last_peered":"2024-09-15T21:02:42.003525+0000","last_clean":"2024-09-15T21:02:42.003525+0000","last_became_active":"2024-09-15T21:02:41.002081+0000","last_became_peered":"2024-09-15T21:02:41.002081+0000","last_unstale":"2024-09-15T21:02:42.003525+0000","last_undegraded":"2024-09-15T21:02:42.003525+0000","last_fullsized":"2024-09-15T21:02:42.003525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:56.145668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003578+0000","last_change":"2024-09-15T21:02:41.005290+0000","last_active":"2024-09-15T21:02:42.003578+0000","last_peered":"2024-09-15T21:02:42.003578+0000","last_clean":"2024-09-15T21:02:42.003578+0000","last_became_active":"2024-09-15T21:02:41.004822+0000","last_became_peered":"2024-09-15T21:02:41.004822+0000","last_unstale":"2024-09-15T21:02:42.003578+0000","last_undegraded":"2024-09-15T21:02:42.003578+0000","last_fullsized":"2024-09-15T21:02:42.003578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:43.163756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003107+0000","last_change":"2024-09-15T21:02:41.002426+0000","last_active":"2024-09-15T21:02:42.003107+0000","last_peered":"2024-09-15T21:02:42.003107+0000","last_clean":"2024-09-15T21:02:42.003107+0000","last_became_active":"2024-09-15T21:02:41.001595+0000","last_became_peered":"2024-09-15T21:02:41.001595+0000","last_unstale":"2024-09-15T21:02:42.003107+0000","last_undegraded":"2024-09-15T21:02:42.003107+0000","last_fullsized":"2024-09-15T21:02:42.003107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:45:31.676166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003636+0000","last_change":"2024-09-15T21:02:41.006060+0000","last_active":"2024-09-15T21:02:42.003636+0000","last_peered":"2024-09-15T21:02:42.003636+0000","last_clean":"2024-09-15T21:02:42.003636+0000","last_became_active":"2024-09-15T21:02:41.005224+0000","last_became_peered":"2024-09-15T21:02:41.005224+0000","last_unstale":"2024-09-15T21:02:42.003636+0000","last_undegraded":"2024-09-15T21:02:42.003636+0000","last_fullsized":"2024-09-15T21:02:42.003636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:50:31.724159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003276+0000","last_change":"2024-09-15T21:02:41.002141+0000","last_active":"2024-09-15T21:02:42.003276+0000","last_peered":"2024-09-15T21:02:42.003276+0000","last_clean":"2024-09-15T21:02:42.003276+0000","last_became_active":"2024-09-15T21:02:41.001768+0000","last_became_peered":"2024-09-15T21:02:41.001768+0000","last_unstale":"2024-09-15T21:02:42.003276+0000","last_undegraded":"2024-09-15T21:02:42.003276+0000","last_fullsized":"2024-09-15T21:02:42.003276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:53.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":122880,"data_stored":35680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":114688,"data_stored":31723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":39663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:02:49.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:02:49.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:02:49.382 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:02:49.393 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-15T21:02:47.900026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":108696,"kb_used_data":560,"kb_used_omap":6,"kb_used_meta":108025,"kb_avail":377378664,"statfs":{"total":386547056640,"available":386435751936,"internally_reserved":0,"allocated":573440,"data_stored":190840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6377,"internal_metadata":110618391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.065006+0000","last_change":"2024-09-15T21:02:41.000187+0000","last_active":"2024-09-15T21:02:42.065006+0000","last_peered":"2024-09-15T21:02:42.065006+0000","last_clean":"2024-09-15T21:02:42.065006+0000","last_became_active":"2024-09-15T21:02:40.999138+0000","last_became_peered":"2024-09-15T21:02:40.999138+0000","last_unstale":"2024-09-15T21:02:42.065006+0000","last_undegraded":"2024-09-15T21:02:42.065006+0000","last_fullsized":"2024-09-15T21:02:42.065006+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:04:39.944817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003329+0000","last_change":"2024-09-15T21:02:41.005604+0000","last_active":"2024-09-15T21:02:42.003329+0000","last_peered":"2024-09-15T21:02:42.003329+0000","last_clean":"2024-09-15T21:02:42.003329+0000","last_became_active":"2024-09-15T21:02:41.004622+0000","last_became_peered":"2024-09-15T21:02:41.004622+0000","last_unstale":"2024-09-15T21:02:42.003329+0000","last_undegraded":"2024-09-15T21:02:42.003329+0000","last_fullsized":"2024-09-15T21:02:42.003329+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:44:46.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.002723+0000","last_change":"2024-09-15T21:02:40.992065+0000","last_active":"2024-09-15T21:02:42.002723+0000","last_peered":"2024-09-15T21:02:42.002723+0000","last_clean":"2024-09-15T21:02:42.002723+0000","last_became_active":"2024-09-15T21:02:40.991678+0000","last_became_peered":"2024-09-15T21:02:40.991678+0000","last_unstale":"2024-09-15T21:02:42.002723+0000","last_undegraded":"2024-09-15T21:02:42.002723+0000","last_fullsized":"2024-09-15T21:02:42.002723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:28:46.175721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003288+0000","last_change":"2024-09-15T21:02:41.002709+0000","last_active":"2024-09-15T21:02:42.003288+0000","last_peered":"2024-09-15T21:02:42.003288+0000","last_clean":"2024-09-15T21:02:42.003288+0000","last_became_active":"2024-09-15T21:02:41.002136+0000","last_became_peered":"2024-09-15T21:02:41.002136+0000","last_unstale":"2024-09-15T21:02:42.003288+0000","last_undegraded":"2024-09-15T21:02:42.003288+0000","last_fullsized":"2024-09-15T21:02:42.003288+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:17:36.601796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.064558+0000","last_change":"2024-09-15T21:02:41.000340+0000","last_active":"2024-09-15T21:02:42.064558+0000","last_peered":"2024-09-15T21:02:42.064558+0000","last_clean":"2024-09-15T21:02:42.064558+0000","last_became_active":"2024-09-15T21:02:40.999945+0000","last_became_peered":"2024-09-15T21:02:40.999945+0000","last_unstale":"2024-09-15T21:02:42.064558+0000","last_undegraded":"2024-09-15T21:02:42.064558+0000","last_fullsized":"2024-09-15T21:02:42.064558+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:51:29.946513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.064666+0000","last_change":"2024-09-15T21:02:40.999021+0000","last_active":"2024-09-15T21:02:42.064666+0000","last_peered":"2024-09-15T21:02:42.064666+0000","last_clean":"2024-09-15T21:02:42.064666+0000","last_became_active":"2024-09-15T21:02:40.998596+0000","last_became_peered":"2024-09-15T21:02:40.998596+0000","last_unstale":"2024-09-15T21:02:42.064666+0000","last_undegraded":"2024-09-15T21:02:42.064666+0000","last_fullsized":"2024-09-15T21:02:42.064666+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:00:42.624139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.070628+0000","last_change":"2024-09-15T21:02:40.993255+0000","last_active":"2024-09-15T21:02:42.070628+0000","last_peered":"2024-09-15T21:02:42.070628+0000","last_clean":"2024-09-15T21:02:42.070628+0000","last_became_active":"2024-09-15T21:02:40.992835+0000","last_became_peered":"2024-09-15T21:02:40.992835+0000","last_unstale":"2024-09-15T21:02:42.070628+0000","last_undegraded":"2024-09-15T21:02:42.070628+0000","last_fullsized":"2024-09-15T21:02:42.070628+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:51:59.355039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003649+0000","last_change":"2024-09-15T21:02:41.005652+0000","last_active":"2024-09-15T21:02:42.003649+0000","last_peered":"2024-09-15T21:02:42.003649+0000","last_clean":"2024-09-15T21:02:42.003649+0000","last_became_active":"2024-09-15T21:02:41.004773+0000","last_became_peered":"2024-09-15T21:02:41.004773+0000","last_unstale":"2024-09-15T21:02:42.003649+0000","last_undegraded":"2024-09-15T21:02:42.003649+0000","last_fullsized":"2024-09-15T21:02:42.003649+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:50.374186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"16'1","reported_seq":17,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.064996+0000","last_change":"2024-09-15T21:02:41.000410+0000","last_active":"2024-09-15T21:02:42.064996+0000","last_peered":"2024-09-15T21:02:42.064996+0000","last_clean":"2024-09-15T21:02:42.064996+0000","last_became_active":"2024-09-15T21:02:40.999792+0000","last_became_peered":"2024-09-15T21:02:40.999792+0000","last_unstale":"2024-09-15T21:02:42.064996+0000","last_undegraded":"2024-09-15T21:02:42.064996+0000","last_fullsized":"2024-09-15T21:02:42.064996+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:59.428303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.002723+0000","last_change":"2024-09-15T21:02:40.992188+0000","last_active":"2024-09-15T21:02:42.002723+0000","last_peered":"2024-09-15T21:02:42.002723+0000","last_clean":"2024-09-15T21:02:42.002723+0000","last_became_active":"2024-09-15T21:02:40.991794+0000","last_became_peered":"2024-09-15T21:02:40.991794+0000","last_unstale":"2024-09-15T21:02:42.002723+0000","last_undegraded":"2024-09-15T21:02:42.002723+0000","last_fullsized":"2024-09-15T21:02:42.002723+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:43:27.853210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.064999+0000","last_change":"2024-09-15T21:02:40.999096+0000","last_active":"2024-09-15T21:02:42.064999+0000","last_peered":"2024-09-15T21:02:42.064999+0000","last_clean":"2024-09-15T21:02:42.064999+0000","last_became_active":"2024-09-15T21:02:40.998660+0000","last_became_peered":"2024-09-15T21:02:40.998660+0000","last_unstale":"2024-09-15T21:02:42.064999+0000","last_undegraded":"2024-09-15T21:02:42.064999+0000","last_fullsized":"2024-09-15T21:02:42.064999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:29:10.390854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003525+0000","last_change":"2024-09-15T21:02:41.002468+0000","last_active":"2024-09-15T21:02:42.003525+0000","last_peered":"2024-09-15T21:02:42.003525+0000","last_clean":"2024-09-15T21:02:42.003525+0000","last_became_active":"2024-09-15T21:02:41.002081+0000","last_became_peered":"2024-09-15T21:02:41.002081+0000","last_unstale":"2024-09-15T21:02:42.003525+0000","last_undegraded":"2024-09-15T21:02:42.003525+0000","last_fullsized":"2024-09-15T21:02:42.003525+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:46:56.145668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003578+0000","last_change":"2024-09-15T21:02:41.005290+0000","last_active":"2024-09-15T21:02:42.003578+0000","last_peered":"2024-09-15T21:02:42.003578+0000","last_clean":"2024-09-15T21:02:42.003578+0000","last_became_active":"2024-09-15T21:02:41.004822+0000","last_became_peered":"2024-09-15T21:02:41.004822+0000","last_unstale":"2024-09-15T21:02:42.003578+0000","last_undegraded":"2024-09-15T21:02:42.003578+0000","last_fullsized":"2024-09-15T21:02:42.003578+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:57:43.163756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003107+0000","last_change":"2024-09-15T21:02:41.002426+0000","last_active":"2024-09-15T21:02:42.003107+0000","last_peered":"2024-09-15T21:02:42.003107+0000","last_clean":"2024-09-15T21:02:42.003107+0000","last_became_active":"2024-09-15T21:02:41.001595+0000","last_became_peered":"2024-09-15T21:02:41.001595+0000","last_unstale":"2024-09-15T21:02:42.003107+0000","last_undegraded":"2024-09-15T21:02:42.003107+0000","last_fullsized":"2024-09-15T21:02:42.003107+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:45:31.676166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003636+0000","last_change":"2024-09-15T21:02:41.006060+0000","last_active":"2024-09-15T21:02:42.003636+0000","last_peered":"2024-09-15T21:02:42.003636+0000","last_clean":"2024-09-15T21:02:42.003636+0000","last_became_active":"2024-09-15T21:02:41.005224+0000","last_became_peered":"2024-09-15T21:02:41.005224+0000","last_unstale":"2024-09-15T21:02:42.003636+0000","last_undegraded":"2024-09-15T21:02:42.003636+0000","last_fullsized":"2024-09-15T21:02:42.003636+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:50:31.724159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-15T21:02:42.003276+0000","last_change":"2024-09-15T21:02:41.002141+0000","last_active":"2024-09-15T21:02:42.003276+0000","last_peered":"2024-09-15T21:02:42.003276+0000","last_clean":"2024-09-15T21:02:42.003276+0000","last_became_active":"2024-09-15T21:02:41.001768+0000","last_became_peered":"2024-09-15T21:02:41.001768+0000","last_unstale":"2024-09-15T21:02:42.003276+0000","last_undegraded":"2024-09-15T21:02:42.003276+0000","last_fullsized":"2024-09-15T21:02:42.003276+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:06:53.898002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":143360,"data_stored":47710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:02:49.395 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-15T21:02:49.395 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T21:02:49.395 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T21:02:49.448 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T21:02:49.448 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T21:02:49.502 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_1 2024-09-15T21:02:49.502 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_1 2024-09-15T21:02:49.552 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T21:02:49.552 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T21:02:49.604 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T21:02:49.604 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T21:02:49.655 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_2 2024-09-15T21:02:49.655 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_2 2024-09-15T21:02:49.705 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T21:02:49.706 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T21:02:49.759 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T21:02:49.759 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T21:02:49.810 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_3 2024-09-15T21:02:49.810 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_3 2024-09-15T21:02:49.860 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T21:02:49.860 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T21:02:49.911 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T21:02:49.911 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T21:02:49.962 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_4 2024-09-15T21:02:49.962 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_4 2024-09-15T21:02:50.012 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T21:02:50.012 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T21:02:50.063 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T21:02:50.063 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T21:02:50.116 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_5 2024-09-15T21:02:50.116 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_5 2024-09-15T21:02:50.166 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T21:02:50.166 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T21:02:50.218 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T21:02:50.218 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T21:02:50.269 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_6 2024-09-15T21:02:50.269 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_6 2024-09-15T21:02:50.319 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T21:02:50.319 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T21:02:50.370 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T21:02:50.370 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T21:02:50.422 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_7 2024-09-15T21:02:50.422 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_7 2024-09-15T21:02:50.474 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T21:02:50.474 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T21:02:50.524 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T21:02:50.525 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T21:02:50.575 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_8 2024-09-15T21:02:50.576 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_8 2024-09-15T21:02:50.625 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T21:02:50.625 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T21:02:50.676 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T21:02:50.676 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T21:02:50.731 INFO:tasks.util.rados:rados -p unique_pool_0 rm existed_9 2024-09-15T21:02:50.731 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 rm existed_9 2024-09-15T21:02:50.781 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 injectargs '--osd-recovery-delay-start 1000 --osd-min-pg-log-entries 100000000' 2024-09-15T21:02:50.901 INFO:teuthology.orchestra.run.smithi181.stdout:{} 2024-09-15T21:02:50.902 INFO:teuthology.orchestra.run.smithi181.stderr:osd_min_pg_log_entries = '' (not observed, change may require restart) osd_recovery_delay_start = '' 2024-09-15T21:02:50.912 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T21:02:50.912 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:02:50.947 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T21:02:50.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2024-09-15T21:02:51.285 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:02:51.282+0000 7f52d8ac4640 -1 mon.c@2(leader).osd e17 definitely_dead 0 2024-09-15T21:02:51.285 INFO:teuthology.orchestra.run.smithi181.stderr:osd.0 is already down. 2024-09-15T21:02:51.298 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T21:02:51.298 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:02:51.327 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T21:02:51.327 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 3 2024-09-15T21:02:51.657 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:02:51.654+0000 7f52d8ac4640 -1 mon.c@2(leader).osd e17 definitely_dead 0 2024-09-15T21:02:52.087 INFO:teuthology.orchestra.run.smithi181.stderr:marked down osd.3. 2024-09-15T21:02:52.102 INFO:tasks.util.rados:rados -p unique_pool_0 put new_1 /etc/group 2024-09-15T21:02:52.102 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_1 /etc/group 2024-09-15T21:02:53.101 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T21:02:53.101 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_1 /etc/group 2024-09-15T21:02:53.156 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T21:02:53.157 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_1 /etc/group 2024-09-15T21:02:53.209 INFO:tasks.util.rados:rados -p unique_pool_0 put new_2 /etc/group 2024-09-15T21:02:53.209 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_2 /etc/group 2024-09-15T21:02:53.260 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T21:02:53.260 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_2 /etc/group 2024-09-15T21:02:53.310 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T21:02:53.311 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_2 /etc/group 2024-09-15T21:02:53.361 INFO:tasks.util.rados:rados -p unique_pool_0 put new_3 /etc/group 2024-09-15T21:02:53.361 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_3 /etc/group 2024-09-15T21:02:53.412 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T21:02:53.412 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_3 /etc/group 2024-09-15T21:02:53.463 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T21:02:53.463 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_3 /etc/group 2024-09-15T21:02:53.514 INFO:tasks.util.rados:rados -p unique_pool_0 put new_4 /etc/group 2024-09-15T21:02:53.514 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_4 /etc/group 2024-09-15T21:02:53.565 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T21:02:53.565 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_4 /etc/group 2024-09-15T21:02:53.615 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T21:02:53.615 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_4 /etc/group 2024-09-15T21:02:53.666 INFO:tasks.util.rados:rados -p unique_pool_0 put new_5 /etc/group 2024-09-15T21:02:53.666 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_5 /etc/group 2024-09-15T21:02:53.716 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T21:02:53.716 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_5 /etc/group 2024-09-15T21:02:53.767 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T21:02:53.767 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_5 /etc/group 2024-09-15T21:02:53.817 INFO:tasks.util.rados:rados -p unique_pool_0 put new_6 /etc/group 2024-09-15T21:02:53.818 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_6 /etc/group 2024-09-15T21:02:53.869 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T21:02:53.869 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_6 /etc/group 2024-09-15T21:02:53.919 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T21:02:53.919 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_6 /etc/group 2024-09-15T21:02:53.971 INFO:tasks.util.rados:rados -p unique_pool_0 put new_7 /etc/group 2024-09-15T21:02:53.971 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_7 /etc/group 2024-09-15T21:02:54.022 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T21:02:54.022 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_7 /etc/group 2024-09-15T21:02:54.072 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T21:02:54.072 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_7 /etc/group 2024-09-15T21:02:54.124 INFO:tasks.util.rados:rados -p unique_pool_0 put new_8 /etc/group 2024-09-15T21:02:54.124 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_8 /etc/group 2024-09-15T21:02:54.177 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T21:02:54.177 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_8 /etc/group 2024-09-15T21:02:54.231 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T21:02:54.232 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_8 /etc/group 2024-09-15T21:02:54.282 INFO:tasks.util.rados:rados -p unique_pool_0 put new_9 /etc/group 2024-09-15T21:02:54.282 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put new_9 /etc/group 2024-09-15T21:02:54.333 INFO:tasks.util.rados:rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T21:02:54.333 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existed_9 /etc/group 2024-09-15T21:02:54.383 INFO:tasks.util.rados:rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T21:02:54.384 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put existing_9 /etc/group 2024-09-15T21:02:54.434 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T21:02:54.434 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:02:54.469 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T21:02:54.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 1 2024-09-15T21:02:54.803 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:02:54.800+0000 7f52d8ac4640 -1 mon.c@2(leader).osd e19 definitely_dead 0 2024-09-15T21:02:55.104 INFO:teuthology.orchestra.run.smithi181.stderr:marked down osd.1. 2024-09-15T21:02:55.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd lost 1 --yes-i-really-mean-it 2024-09-15T21:02:56.112 INFO:teuthology.orchestra.run.smithi181.stderr:marked osd lost in epoch 20 2024-09-15T21:02:56.127 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T21:02:56.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T21:02:56.131 INFO:tasks.ceph.osd.0:Started 2024-09-15T21:02:56.131 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T21:02:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:02:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:02:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:02:56.231 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-15T21:02:56.232 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:02:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:02:56.234 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-15T21:02:57.302 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:02:57.299+0000 7f347381b640 -1 Falling back to public interface 2024-09-15T21:03:01.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T21:03:01.322 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:03:01.319+0000 7f347381b640 -1 osd.0 16 log_to_monitors true 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs debug_inject_read_zeros 2024-09-15T21:03:01.346 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs files list 2024-09-15T21:03:01.347 INFO:teuthology.orchestra.run.smithi181.stderr:bluefs stats 2024-09-15T21:03:01.347 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator dump block 2024-09-15T21:03:01.347 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator fragmentation block 2024-09-15T21:03:01.347 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:03:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:03:01.349 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-15T21:03:02.195 INFO:tasks.ceph.osd.0.smithi181.stderr:2024-09-15T21:03:02.192+0000 7f346af54640 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:03:06.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-15T21:03:06.466 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:06.466 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-15T21:03:06.466 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-15T21:03:06.466 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:06.475 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.0 to be up 2024-09-15T21:03:06.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:03:06.757 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:06.758 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":24,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:03:04.173309+0000","last_up_change":"2024-09-15T21:03:03.171948+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T21:02:37.977251+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":23,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6809","nonce":2996802086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6811","nonce":2996802086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6815","nonce":2996802086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6813","nonce":2996802086}]},"public_addr":"172.21.15.181:6809/2996802086","cluster_addr":"172.21.15.181:6811/2996802086","heartbeat_back_addr":"172.21.15.181:6815/2996802086","heartbeat_front_addr":"172.21.15.181:6813/2996802086","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":17,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6825","nonce":2479280537}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6827","nonce":2479280537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6831","nonce":2479280537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":2479280537},{"type":"v1","addr":"172.21.15.181:6829","nonce":2479280537}]},"public_addr":"172.21.15.181:6825/2479280537","cluster_addr":"172.21.15.181:6827/2479280537","heartbeat_back_addr":"172.21.15.181:6831/2479280537","heartbeat_front_addr":"172.21.15.181:6829/2479280537","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:03:06.770 INFO:tasks.ec_lost_unfound.ceph_manager:osd.0 is up 2024-09-15T21:03:06.770 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T21:03:06.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T21:03:06.773 INFO:tasks.ceph.osd.3:Started 2024-09-15T21:03:06.773 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T21:03:06.871 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-15T21:03:06.872 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:03:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:03:06.874 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-15T21:03:08.192 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:03:08.188+0000 7f52db2c9640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 14 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T21:03:08.434 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:03:08.430+0000 7f90562b4640 -1 Falling back to public interface 2024-09-15T21:03:11.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T21:03:11.982 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator dump block 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator fragmentation block 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:bluestore allocator score block 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-15T21:03:11.983 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:03:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:03:11.985 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-15T21:03:12.212 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:03:12.209+0000 7f90562b4640 -1 osd.3 17 log_to_monitors true 2024-09-15T21:03:12.797 INFO:tasks.ceph.osd.3.smithi181.stderr:2024-09-15T21:03:12.794+0000 7f904a062640 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:03:16.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-15T21:03:17.101 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:17.101 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [], 2024-09-15T21:03:17.102 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 0 2024-09-15T21:03:17.102 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:17.110 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.3 to be up 2024-09-15T21:03:17.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:03:17.396 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":27,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:03:14.791306+0000","last_up_change":"2024-09-15T21:03:13.789368+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T21:02:37.977251+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":26,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6809","nonce":2996802086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6811","nonce":2996802086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6815","nonce":2996802086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6813","nonce":2996802086}]},"public_addr":"172.21.15.181:6809/2996802086","cluster_addr":"172.21.15.181:6811/2996802086","heartbeat_back_addr":"172.21.15.181:6815/2996802086","heartbeat_front_addr":"172.21.15.181:6813/2996802086","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":18,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6809","nonce":2843583490}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6811","nonce":2843583490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6815","nonce":2843583490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2843583490},{"type":"v1","addr":"172.21.15.181:6813","nonce":2843583490}]},"public_addr":"172.21.15.181:6809/2843583490","cluster_addr":"172.21.15.181:6811/2843583490","heartbeat_back_addr":"172.21.15.181:6815/2843583490","heartbeat_front_addr":"172.21.15.181:6813/2843583490","state":["exists"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":26,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6817","nonce":2345571741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6819","nonce":2345571741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6823","nonce":2345571741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6821","nonce":2345571741}]},"public_addr":"172.21.15.181:6817/2345571741","cluster_addr":"172.21.15.181:6819/2345571741","heartbeat_back_addr":"172.21.15.181:6823/2345571741","heartbeat_front_addr":"172.21.15.181:6821/2345571741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:03:17.411 INFO:tasks.ec_lost_unfound.ceph_manager:osd.3 is up 2024-09-15T21:03:17.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:17.412 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:17.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:17.551 INFO:teuthology.orchestra.run.smithi181.stdout:98784247813 2024-09-15T21:03:17.551 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:17.557 INFO:teuthology.orchestra.run.smithi181.stdout:111669149699 2024-09-15T21:03:17.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:17.559 INFO:teuthology.orchestra.run.smithi181.stdout:38654705678 2024-09-15T21:03:17.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:17.854 INFO:teuthology.orchestra.run.smithi181.stdout:98784247811 2024-09-15T21:03:17.867 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247813 got 98784247811 for osd.0 2024-09-15T21:03:17.897 INFO:teuthology.orchestra.run.smithi181.stdout:38654705676 2024-09-15T21:03:17.899 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-15T21:03:17.911 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705676 for osd.2 2024-09-15T21:03:17.913 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 1 for osd.3 2024-09-15T21:03:18.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:18.912 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:18.913 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:19.185 INFO:teuthology.orchestra.run.smithi181.stdout:98784247813 2024-09-15T21:03:19.198 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247813 got 98784247813 for osd.0 2024-09-15T21:03:19.198 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:19.216 INFO:teuthology.orchestra.run.smithi181.stdout:111669149699 2024-09-15T21:03:19.221 INFO:teuthology.orchestra.run.smithi181.stdout:38654705678 2024-09-15T21:03:19.232 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149699 got 111669149699 for osd.3 2024-09-15T21:03:19.232 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:19.236 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705678 got 38654705678 for osd.2 2024-09-15T21:03:19.236 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:19.236 INFO:tasks.ec_lost_unfound.ceph_manager:waiting till active 2024-09-15T21:03:19.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:03:19.557 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:19.558 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:03:19.569 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-15T21:03:17.906150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":110116,"kb_used_data":1032,"kb_used_omap":11,"kb_used_meta":108980,"kb_avail":377377244,"statfs":{"total":386547056640,"available":386434297856,"internally_reserved":0,"allocated":1056768,"data_stored":535403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11992,"internal_metadata":111595816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001518"},"pg_stats":[{"pgid":"1.f","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-15T21:03:14.815031+0000","last_change":"2024-09-15T21:03:14.814722+0000","last_active":"2024-09-15T21:03:14.815031+0000","last_peered":"2024-09-15T21:03:14.815031+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:14.815031+0000","last_undegraded":"2024-09-15T21:03:14.815031+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:11:10.422591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.820530+0000","last_change":"2024-09-15T21:03:14.820897+0000","last_active":"2024-09-15T21:03:14.820530+0000","last_peered":"2024-09-15T21:03:14.820530+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:14.820530+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"19'3","reported_seq":56,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.824147+0000","last_change":"2024-09-15T21:03:14.824381+0000","last_active":"2024-09-15T21:03:14.824147+0000","last_peered":"2024-09-15T21:03:14.824147+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:14.824147+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'4","reported_seq":44,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.821844+0000","last_change":"2024-09-15T21:03:14.822168+0000","last_active":"2024-09-15T21:03:14.821844+0000","last_peered":"2024-09-15T21:03:14.821844+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:14.821844+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"19'3","reported_seq":32,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.827838+0000","last_change":"2024-09-15T21:03:14.828116+0000","last_active":"2024-09-15T21:03:14.827838+0000","last_peered":"2024-09-15T21:03:14.827838+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:14.827838+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":29,"reported_epoch":27,"state":"active+undersized","last_fresh":"2024-09-15T21:03:14.815105+0000","last_change":"2024-09-15T21:03:14.814704+0000","last_active":"2024-09-15T21:03:14.815105+0000","last_peered":"2024-09-15T21:03:14.815105+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:14.815105+0000","last_undegraded":"2024-09-15T21:03:14.815105+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:23.850087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"19'6","reported_seq":46,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.818847+0000","last_change":"2024-09-15T21:03:14.819174+0000","last_active":"2024-09-15T21:03:14.818847+0000","last_peered":"2024-09-15T21:03:14.818847+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:14.818847+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.824704+0000","last_change":"2024-09-15T21:03:14.825039+0000","last_active":"2024-09-15T21:03:14.824704+0000","last_peered":"2024-09-15T21:03:14.824704+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:14.824704+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"19'9","reported_seq":37,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.825291+0000","last_change":"2024-09-15T21:03:14.825593+0000","last_active":"2024-09-15T21:03:14.825291+0000","last_peered":"2024-09-15T21:03:14.825291+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:14.825291+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"19'2","reported_seq":54,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.826171+0000","last_change":"2024-09-15T21:03:14.826405+0000","last_active":"2024-09-15T21:03:14.826171+0000","last_peered":"2024-09-15T21:03:14.826171+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:14.826171+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"19'5","reported_seq":33,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.822466+0000","last_change":"2024-09-15T21:03:14.822934+0000","last_active":"2024-09-15T21:03:14.822466+0000","last_peered":"2024-09-15T21:03:14.822466+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:14.822466+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20370,"num_objects":3,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"19'1","reported_seq":17,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.819817+0000","last_change":"2024-09-15T21:03:14.819817+0000","last_active":"2024-09-15T21:03:14.819817+0000","last_peered":"2024-09-15T21:03:14.819817+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:14.819817+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"19'1","reported_seq":54,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.817500+0000","last_change":"2024-09-15T21:03:14.817864+0000","last_active":"2024-09-15T21:03:14.817500+0000","last_peered":"2024-09-15T21:03:14.817500+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:14.817500+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"19'4","reported_seq":32,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.827245+0000","last_change":"2024-09-15T21:03:14.827245+0000","last_active":"2024-09-15T21:03:14.827245+0000","last_peered":"2024-09-15T21:03:14.827245+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:14.827245+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"19'3","reported_seq":57,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.821253+0000","last_change":"2024-09-15T21:03:14.821537+0000","last_active":"2024-09-15T21:03:14.821253+0000","last_peered":"2024-09-15T21:03:14.821253+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:14.821253+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"19'5","reported_seq":45,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:14.823507+0000","last_change":"2024-09-15T21:03:14.823853+0000","last_active":"2024-09-15T21:03:14.823507+0000","last_peered":"2024-09-15T21:03:14.823507+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:14.823507+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":190120,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":82,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":270336,"data_stored":128431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":28176080},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":344064,"data_stored":202159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":76382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":264,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":270336,"data_stored":128431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":28110542},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:03:19.571 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:03:19.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:19.922 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":65,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726434193,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":112758784,"bytes_avail":386434297856,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-15T21:02:14.253593+0000","services":{}},"progress_events":{}} 2024-09-15T21:03:19.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c31fa425-5a82-4117-b11e-63b4944cd2c3', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 65, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726434193, 'num_in_osds': 4, 'osd_in_since': 1726434136, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 190120, 'bytes_used': 112758784, 'bytes_avail': 386434297856, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-15T21:02:14:250322+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 1, 'modified': '2024-09-15T21:02:14.253593+0000', 'services': {}}, 'progress_events': {}} 2024-09-15T21:03:19.935 INFO:tasks.ec_lost_unfound.ceph_manager:active! 2024-09-15T21:03:19.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:19.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:19.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:20.072 INFO:teuthology.orchestra.run.smithi181.stdout:111669149700 2024-09-15T21:03:20.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:20.091 INFO:teuthology.orchestra.run.smithi181.stdout:98784247814 2024-09-15T21:03:20.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:20.093 INFO:teuthology.orchestra.run.smithi181.stdout:38654705679 2024-09-15T21:03:20.093 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:20.378 INFO:teuthology.orchestra.run.smithi181.stdout:111669149699 2024-09-15T21:03:20.391 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-15T21:03:20.464 INFO:teuthology.orchestra.run.smithi181.stdout:98784247813 2024-09-15T21:03:20.552 INFO:teuthology.orchestra.run.smithi181.stdout:38654705678 2024-09-15T21:03:20.553 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-15T21:03:20.554 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2024-09-15T21:03:21.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:21.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:21.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:21.700 INFO:teuthology.orchestra.run.smithi181.stdout:111669149699 2024-09-15T21:03:21.713 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149699 for osd.3 2024-09-15T21:03:21.882 INFO:teuthology.orchestra.run.smithi181.stdout:38654705678 2024-09-15T21:03:21.883 INFO:teuthology.orchestra.run.smithi181.stdout:98784247813 2024-09-15T21:03:21.896 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705678 for osd.2 2024-09-15T21:03:21.896 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247813 for osd.0 2024-09-15T21:03:22.714 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:22.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:22.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:23.022 INFO:teuthology.orchestra.run.smithi181.stdout:111669149700 2024-09-15T21:03:23.036 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149700 got 111669149700 for osd.3 2024-09-15T21:03:23.036 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:23.193 INFO:teuthology.orchestra.run.smithi181.stdout:38654705680 2024-09-15T21:03:23.194 INFO:teuthology.orchestra.run.smithi181.stdout:98784247814 2024-09-15T21:03:23.207 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705679 got 38654705680 for osd.2 2024-09-15T21:03:23.208 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:23.208 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247814 got 98784247814 for osd.0 2024-09-15T21:03:23.208 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:23.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:03:23.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:23.561 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"27/28 objects unfound (96.429%)","count":27},"muted":false},"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 14 pgs recovery_unfound","count":14},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":69,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":27,"num_osds":4,"num_up_osds":3,"osd_up_since":1726434193,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_unfound+undersized+degraded","count":14},{"state_name":"active+undersized","count":2}],"num_pgs":16,"num_pools":1,"num_objects":28,"data_bytes":190120,"bytes_used":112758784,"bytes_avail":386434297856,"bytes_total":386547056640,"degraded_objects":82,"degraded_total":112,"degraded_ratio":0.7321428571428571,"unfound_objects":27,"unfound_total":28,"unfound_ratio":0.9642857142857143},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-15T21:03:21.907073+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:03:23.574 INFO:tasks.ec_lost_unfound.ceph_manager:{'fsid': 'c31fa425-5a82-4117-b11e-63b4944cd2c3', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '27/28 objects unfound (96.429%)', 'count': 27}, 'muted': False}, 'OSD_DOWN': {'severity': 'HEALTH_WARN', 'summary': {'message': '1 osds down', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 14 pgs recovery_unfound', 'count': 14}, 'muted': False}, 'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 82/112 objects degraded (73.214%), 14 pgs degraded', 'count': 14}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 69, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 27, 'num_osds': 4, 'num_up_osds': 3, 'osd_up_since': 1726434193, 'num_in_osds': 4, 'osd_in_since': 1726434136, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_unfound+undersized+degraded', 'count': 14}, {'state_name': 'active+undersized', 'count': 2}], 'num_pgs': 16, 'num_pools': 1, 'num_objects': 28, 'data_bytes': 190120, 'bytes_used': 112758784, 'bytes_avail': 386434297856, 'bytes_total': 386547056640, 'degraded_objects': 82, 'degraded_total': 112, 'degraded_ratio': 0.7321428571428571, 'unfound_objects': 27, 'unfound_total': 28, 'unfound_ratio': 0.9642857142857143}, 'fsmap': {'epoch': 1, 'btime': '2024-09-15T21:02:14:250322+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 2, 'modified': '2024-09-15T21:03:21.907073+0000', 'services': {'osd': {'daemons': {'summary': '', '0': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '1': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '2': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}, '3': {'start_epoch': 0, 'start_stamp': '0.000000', 'gid': 0, 'addr': '(unrecognized address family 0)/0', 'metadata': {}, 'task_status': {}}}}}}, 'progress_events': {}} 2024-09-15T21:03:23.574 INFO:tasks.ec_lost_unfound:there are 27 unfound objects 2024-09-15T21:03:23.575 DEBUG:teuthology.orchestra.run.smithi181:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.admin -b 4096 -p unique_pool_0 -t 20 bench 240 write' 2024-09-15T21:03:23.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stderr:INFO: op_size has been rounded to 8192 2024-09-15T21:03:23.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:hints = 1 2024-09-15T21:03:23.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Maintaining 20 concurrent writes of 8192 bytes to objects of size 8192 for up to 240 seconds or 0 objects 2024-09-15T21:03:23.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Object prefix: benchmark_data_smithi181_43982 2024-09-15T21:03:23.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:03:23.627 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 0 0 0 0 0 0 - 0 2024-09-15T21:03:24.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 1 20 1715 1695 13.2413 13.2422 0.0127064 0.0117016 2024-09-15T21:03:25.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 2 20 3380 3360 13.1241 13.0078 0.0187731 0.0118224 2024-09-15T21:03:26.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 3 20 4919 4899 12.7569 12.0234 0.0106643 0.0122142 2024-09-15T21:03:27.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 4 20 6498 6478 12.6514 12.3359 0.0192406 0.01233 2024-09-15T21:03:28.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 5 20 7993 7973 12.4569 11.6797 0.010454 0.0125196 2024-09-15T21:03:29.626 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 6 20 9442 9422 12.2674 11.3203 0.0110896 0.0127111 2024-09-15T21:03:30.627 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 7 20 10852 10832 12.0884 11.0156 0.0100628 0.0129115 2024-09-15T21:03:31.627 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 8 20 12189 12169 11.8828 10.4453 0.0110869 0.0131306 2024-09-15T21:03:32.627 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 9 19 13507 13488 11.7074 10.3047 0.0155455 0.0133307 2024-09-15T21:03:33.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:03:33.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 10 20 14736 14716 11.495 9.59375 0.0099726 0.01358 2024-09-15T21:03:34.130 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:34.131 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:03:34.151 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-15T21:03:33.910775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":103106216,"num_objects":12591,"num_object_clones":0,"num_object_copies":50364,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":12645,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":12591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12618,"num_write_kb":100826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12618,"ondisk_log_size":12618,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":336524,"kb_used_data":142796,"kb_used_omap":11,"kb_used_meta":193652,"kb_avail":377150836,"statfs":{"total":386547056640,"available":386202456064,"internally_reserved":0,"allocated":146223104,"data_stored":145701739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11992,"internal_metadata":198299944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,13,6,10,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":102916096,"num_objects":12563,"num_object_clones":0,"num_object_copies":50252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12563,"num_write_kb":100504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003788"},"pg_stats":[{"pgid":"1.f","version":"27'811","reported_seq":840,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:32.250079+0000","last_change":"2024-09-15T21:03:23.636949+0000","last_active":"2024-09-15T21:03:32.250079+0000","last_peered":"2024-09-15T21:03:32.250079+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:32.250079+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:11:10.422591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6643712,"num_objects":811,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":811,"num_write_kb":6488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'812","reported_seq":852,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.167122+0000","last_change":"2024-09-15T21:03:14.820897+0000","last_active":"2024-09-15T21:03:32.167122+0000","last_peered":"2024-09-15T21:03:32.167122+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:32.167122+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":812,"log_dups_size":0,"ondisk_log_size":812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6632716,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":814,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":812,"num_write_kb":6492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":808},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'698","reported_seq":751,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:30.915492+0000","last_change":"2024-09-15T21:03:14.824381+0000","last_active":"2024-09-15T21:03:30.915492+0000","last_peered":"2024-09-15T21:03:30.915492+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:30.915492+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5700230,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":698,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":698,"num_write_kb":5574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":695},{"shards":"2(0)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'813","reported_seq":853,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.137701+0000","last_change":"2024-09-15T21:03:14.822168+0000","last_active":"2024-09-15T21:03:32.137701+0000","last_peered":"2024-09-15T21:03:32.137701+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:32.137701+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":813,"log_dups_size":0,"ondisk_log_size":813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6640908,"num_objects":811,"num_object_clones":0,"num_object_copies":3244,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":815,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":813,"num_write_kb":6493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":809},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'825","reported_seq":854,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.254779+0000","last_change":"2024-09-15T21:03:14.828116+0000","last_active":"2024-09-15T21:03:32.254779+0000","last_peered":"2024-09-15T21:03:32.254779+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:32.254779+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6747404,"num_objects":824,"num_object_clones":0,"num_object_copies":3296,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":828,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":825,"num_write_kb":6597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":822},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'810","reported_seq":839,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:32.267345+0000","last_change":"2024-09-15T21:03:23.649982+0000","last_active":"2024-09-15T21:03:32.267345+0000","last_peered":"2024-09-15T21:03:32.267345+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:32.267345+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":810,"log_dups_size":0,"ondisk_log_size":810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:23.850087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6635520,"num_objects":810,"num_object_clones":0,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":810,"num_write_kb":6480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'849","reported_seq":889,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.165188+0000","last_change":"2024-09-15T21:03:14.819174+0000","last_active":"2024-09-15T21:03:32.165188+0000","last_peered":"2024-09-15T21:03:32.165188+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:32.165188+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6919436,"num_objects":845,"num_object_clones":0,"num_object_copies":3380,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":849,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":849,"num_write_kb":6772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":843},{"shards":"2(3)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'835","reported_seq":875,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.179699+0000","last_change":"2024-09-15T21:03:14.825039+0000","last_active":"2024-09-15T21:03:32.179699+0000","last_peered":"2024-09-15T21:03:32.179699+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:32.179699+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6812940,"num_objects":832,"num_object_clones":0,"num_object_copies":3328,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":836,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":6668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":830},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'843","reported_seq":871,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.271672+0000","last_change":"2024-09-15T21:03:14.825593+0000","last_active":"2024-09-15T21:03:32.271672+0000","last_peered":"2024-09-15T21:03:32.271672+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:32.271672+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":843,"log_dups_size":0,"ondisk_log_size":843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6859288,"num_objects":838,"num_object_clones":0,"num_object_copies":3352,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":844,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":843,"num_write_kb":6721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":835},{"shards":"2(2)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'700","reported_seq":752,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:30.915666+0000","last_change":"2024-09-15T21:03:14.826405+0000","last_active":"2024-09-15T21:03:30.915666+0000","last_peered":"2024-09-15T21:03:30.915666+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:30.915666+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":700,"log_dups_size":0,"ondisk_log_size":700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5731596,"num_objects":700,"num_object_clones":0,"num_object_copies":2800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":704,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":700,"num_write_kb":5598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":698},{"shards":"2(0)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'819","reported_seq":847,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.220686+0000","last_change":"2024-09-15T21:03:14.822934+0000","last_active":"2024-09-15T21:03:32.220686+0000","last_peered":"2024-09-15T21:03:32.220686+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:32.220686+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6688658,"num_objects":817,"num_object_clones":0,"num_object_copies":3268,"num_objects_missing_on_primary":3,"num_objects_missing":3,"num_objects_degraded":823,"num_objects_misplaced":0,"num_objects_unfound":3,"num_objects_dirty":817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":6540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":814},{"shards":"2(3)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'780","reported_seq":796,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.282261+0000","last_change":"2024-09-15T21:03:14.819817+0000","last_active":"2024-09-15T21:03:32.282261+0000","last_peered":"2024-09-15T21:03:32.282261+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:32.282261+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":780,"log_dups_size":0,"ondisk_log_size":780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6388358,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":780,"num_write_kb":6239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":779},{"shards":"2(2)","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'699","reported_seq":752,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:30.887236+0000","last_change":"2024-09-15T21:03:14.817864+0000","last_active":"2024-09-15T21:03:30.887236+0000","last_peered":"2024-09-15T21:03:30.887236+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:30.887236+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5724806,"num_objects":699,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":701,"num_objects_misplaced":0,"num_objects_unfound":1,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":699,"num_write_kb":5591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":698},{"shards":"2(1)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'790","reported_seq":818,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.262047+0000","last_change":"2024-09-15T21:03:14.827245+0000","last_active":"2024-09-15T21:03:32.262047+0000","last_peered":"2024-09-15T21:03:32.262047+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:32.262047+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6452492,"num_objects":788,"num_object_clones":0,"num_object_copies":3152,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":792,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":790,"num_write_kb":6316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":786},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'695","reported_seq":749,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:30.898199+0000","last_change":"2024-09-15T21:03:14.821537+0000","last_active":"2024-09-15T21:03:30.898199+0000","last_peered":"2024-09-15T21:03:30.898199+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:30.898199+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5682444,"num_objects":694,"num_object_clones":0,"num_object_copies":2776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":698,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":5557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":692},{"shards":"2(1)","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'839","reported_seq":879,"reported_epoch":27,"state":"active+recovery_unfound+undersized+degraded","last_fresh":"2024-09-15T21:03:32.142748+0000","last_change":"2024-09-15T21:03:14.823853+0000","last_active":"2024-09-15T21:03:32.142748+0000","last_peered":"2024-09-15T21:03:32.142748+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:32.142748+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":839,"log_dups_size":0,"ondisk_log_size":839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6845708,"num_objects":836,"num_object_clones":0,"num_object_copies":3344,"num_objects_missing_on_primary":2,"num_objects_missing":2,"num_objects_degraded":840,"num_objects_misplaced":0,"num_objects_unfound":2,"num_objects_dirty":836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":839,"num_write_kb":6700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":834},{"shards":"2(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":103106216,"num_objects":12591,"num_object_clones":0,"num_object_copies":50364,"num_objects_missing_on_primary":21,"num_objects_missing":21,"num_objects_degraded":12645,"num_objects_misplaced":0,"num_objects_unfound":27,"num_objects_dirty":12591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12618,"num_write_kb":100826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":152952832,"data_stored":152952832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12618,"ondisk_log_size":12618,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":114464,"kb_used_data":50832,"kb_used_omap":4,"kb_used_meta":63611,"kb_avail":94257376,"statfs":{"total":96636764160,"available":96519553024,"internally_reserved":0,"allocated":52051968,"data_stored":51910063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":65138384},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":88828,"kb_used_data":43756,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94283012,"statfs":{"total":96636764160,"available":96545804288,"internally_reserved":0,"allocated":44806144,"data_stored":44664239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":76382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106040,"kb_used_data":48040,"kb_used_omap":4,"kb_used_meta":57979,"kb_avail":94265800,"statfs":{"total":96636764160,"available":96528179200,"internally_reserved":0,"allocated":49192960,"data_stored":49051055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":59371214},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,4,6,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53100544,"data_stored":53100544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":46223360,"data_stored":46223360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53587968,"data_stored":53587968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:03:34.153 INFO:tasks.ec_lost_unfound:no unfound in 1.f 2024-09-15T21:03:34.153 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.e state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:34.153 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.e list_unfound '{}' 2024-09-15T21:03:34.379 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:34.380 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_4", 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1224592654, 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'1", 2024-09-15T21:03:34.381 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_6", 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3101443614, 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:34.382 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'2", 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:34.383 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:34.384 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:34.399 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_4', 'key': '', 'snapid': -2, 'hash': 1224592654, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'existing_6', 'key': '', 'snapid': -2, 'hash': 3101443614, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:34.399 INFO:tasks.ec_lost_unfound:reverting unfound in 1.e 2024-09-15T21:03:34.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.e mark_unfound_lost delete 2024-09-15T21:03:34.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 11 20 15777 15757 11.1894 8.13281 0.022222 0.0139463 2024-09-15T21:03:34.665 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:34.688 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.d state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:34.688 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.d list_unfound '{}' 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:34.907 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 1, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 1, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_7", 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 2358579165, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:34.908 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "2(0)" 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:34.909 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(1)", 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:34.910 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:34.932 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'existed_7', 'key': '', 'snapid': -2, 'hash': 2358579165, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-15T21:03:34.932 INFO:tasks.ec_lost_unfound:reverting unfound in 1.d 2024-09-15T21:03:34.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.d mark_unfound_lost delete 2024-09-15T21:03:35.182 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T21:03:35.206 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.0 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:35.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.0 list_unfound '{}' 2024-09-15T21:03:35.451 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:35.451 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.452 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_5", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 4248945696, 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:35.453 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_5", 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 4212789968, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.454 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.455 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:35.456 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:35.456 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:35.456 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:35.456 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:35.456 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:35.472 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_5', 'key': '', 'snapid': -2, 'hash': 4248945696, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_5', 'key': '', 'snapid': -2, 'hash': 4212789968, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:35.472 INFO:tasks.ec_lost_unfound:reverting unfound in 1.0 2024-09-15T21:03:35.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.0 mark_unfound_lost delete 2024-09-15T21:03:35.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 12 20 16859 16839 10.9613 8.45312 0.0134362 0.0142406 2024-09-15T21:03:35.719 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:35.740 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.1 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:35.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.1 list_unfound '{}' 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.967 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_3", 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 459768081, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:35.968 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'1", 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_3", 2024-09-15T21:03:35.969 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 480221105, 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'2", 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:35.970 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(1)", 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:35.971 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:35.985 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_3', 'key': '', 'snapid': -2, 'hash': 459768081, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(3)']}, {'oid': {'oid': 'new_3', 'key': '', 'snapid': -2, 'hash': 480221105, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(1)', 'status': 'osd is down'}]} 2024-09-15T21:03:35.985 INFO:tasks.ec_lost_unfound:reverting unfound in 1.1 2024-09-15T21:03:35.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.1 mark_unfound_lost delete 2024-09-15T21:03:36.228 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:36.253 INFO:tasks.ec_lost_unfound:no unfound in 1.c 2024-09-15T21:03:36.253 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.2 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:36.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.2 list_unfound '{}' 2024-09-15T21:03:36.473 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:36.473 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:36.473 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:36.473 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.473 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_6", 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.474 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1708673474, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'5", 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.475 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_9", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 831402034, 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'6", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:36.476 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(2)", 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:36.477 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:36.478 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:36.502 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_6', 'key': '', 'snapid': -2, 'hash': 1708673474, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'existed_9', 'key': '', 'snapid': -2, 'hash': 831402034, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'6", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-15T21:03:36.502 INFO:tasks.ec_lost_unfound:reverting unfound in 1.2 2024-09-15T21:03:36.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.2 mark_unfound_lost delete 2024-09-15T21:03:36.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 13 20 17928 17908 10.7605 8.35156 0.0231145 0.0145104 2024-09-15T21:03:36.751 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:36.771 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.3 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:36.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.3 list_unfound '{}' 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:36.996 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_8", 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1901787523, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.997 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'5", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'3", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_1", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:36.998 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3122379363, 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:36.999 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:37.000 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:37.001 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:37.001 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:37.022 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_8', 'key': '', 'snapid': -2, 'hash': 1901787523, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "16'3", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_1', 'key': '', 'snapid': -2, 'hash': 3122379363, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:37.022 INFO:tasks.ec_lost_unfound:reverting unfound in 1.3 2024-09-15T21:03:37.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.3 mark_unfound_lost delete 2024-09-15T21:03:37.267 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:37.289 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.4 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:37.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.4 list_unfound '{}' 2024-09-15T21:03:37.516 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 3, 2024-09-15T21:03:37.517 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 3, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_4", 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 873891716, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'8", 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:37.518 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_7", 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3996155460, 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:37.519 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'9", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'6", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_2", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:37.520 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1767667508, 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'7", 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:37.521 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(3)", 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:37.522 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:37.534 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_4', 'key': '', 'snapid': -2, 'hash': 873891716, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'8", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_7', 'key': '', 'snapid': -2, 'hash': 3996155460, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'9", 'have': "16'6", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existed_2', 'key': '', 'snapid': -2, 'hash': 1767667508, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'7", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-15T21:03:37.534 INFO:tasks.ec_lost_unfound:reverting unfound in 1.4 2024-09-15T21:03:37.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.4 mark_unfound_lost delete 2024-09-15T21:03:37.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 14 20 18970 18950 10.5733 8.14062 0.0207395 0.0147611 2024-09-15T21:03:37.790 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-15T21:03:37.809 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.5 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:37.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.5 list_unfound '{}' 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.049 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_1", 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 26281365, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'1", 2024-09-15T21:03:38.050 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "2(0)" 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_2", 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3196340981, 2024-09-15T21:03:38.051 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'2", 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: "2(0)" 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:38.052 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(3)", 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:38.053 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:38.072 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'new_1', 'key': '', 'snapid': -2, 'hash': 26281365, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}, {'oid': {'oid': 'new_2', 'key': '', 'snapid': -2, 'hash': 3196340981, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(0)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(3)', 'status': 'osd is down'}]} 2024-09-15T21:03:38.072 INFO:tasks.ec_lost_unfound:reverting unfound in 1.5 2024-09-15T21:03:38.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.5 mark_unfound_lost delete 2024-09-15T21:03:38.344 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:38.366 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.6 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:38.366 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.6 list_unfound '{}' 2024-09-15T21:03:38.402 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:03:38.399+0000 7f52db2c9640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 9 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T21:03:38.595 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:38.595 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 3, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 3, 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:38.596 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_8", 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1861424966, 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'5", 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:38.597 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_6", 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3910131350, 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.598 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_4", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:38.599 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 2787839990, 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: "2(3)" 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:38.600 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(2)", 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:38.601 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:38.619 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 3, 'num_unfound': 3, 'objects': [{'oid': {'oid': 'existed_8', 'key': '', 'snapid': -2, 'hash': 1861424966, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_6', 'key': '', 'snapid': -2, 'hash': 3910131350, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}, {'oid': {'oid': 'new_4', 'key': '', 'snapid': -2, 'hash': 2787839990, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(3)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(2)', 'status': 'osd is down'}]} 2024-09-15T21:03:38.619 INFO:tasks.ec_lost_unfound:reverting unfound in 1.6 2024-09-15T21:03:38.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.6 mark_unfound_lost delete 2024-09-15T21:03:38.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 15 20 20019 19999 10.4148 8.19531 0.0189905 0.0149909 2024-09-15T21:03:38.885 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 3 objects unfound and apparently lost marking 2024-09-15T21:03:38.908 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.7 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:38.908 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.7 list_unfound '{}' 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:39.156 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 1, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 1, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_7", 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 3761017527, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:39.157 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'1", 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:39.158 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:39.159 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:39.159 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:39.181 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_7', 'key': '', 'snapid': -2, 'hash': 3761017527, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:39.182 INFO:tasks.ec_lost_unfound:reverting unfound in 1.7 2024-09-15T21:03:39.182 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.7 mark_unfound_lost delete 2024-09-15T21:03:39.419 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T21:03:39.445 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.8 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:39.445 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.8 list_unfound '{}' 2024-09-15T21:03:39.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 16 20 21038 21018 10.2614 7.96094 0.0225172 0.0152127 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:39.674 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 1, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 1, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_8", 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:39.675 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 2730327960, 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'1", 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: "2(1)" 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:39.676 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:39.677 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:39.698 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'new_8', 'key': '', 'snapid': -2, 'hash': 2730327960, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'1", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:39.698 INFO:tasks.ec_lost_unfound:reverting unfound in 1.8 2024-09-15T21:03:39.698 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.8 mark_unfound_lost delete 2024-09-15T21:03:39.949 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 1 objects unfound and apparently lost marking 2024-09-15T21:03:39.972 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.9 state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:39.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.9 list_unfound '{}' 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:40.205 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_5", 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:40.206 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 2998549577, 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'2", 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.207 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_2", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 4032093161, 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'1", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:40.208 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:40.209 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:40.227 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_5', 'key': '', 'snapid': -2, 'hash': 2998549577, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'2", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_2', 'key': '', 'snapid': -2, 'hash': 4032093161, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:40.227 INFO:tasks.ec_lost_unfound:reverting unfound in 1.9 2024-09-15T21:03:40.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.9 mark_unfound_lost delete 2024-09-15T21:03:40.471 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:40.497 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.a state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:40.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.a list_unfound '{}' 2024-09-15T21:03:40.628 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 17 20 22034 22014 10.1154 7.78125 0.0240123 0.0154328 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.722 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_9", 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 1217469130, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.723 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'3", 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'1", 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "2(1)" 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:40.724 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "new_9", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 170713514, 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'2", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:40.725 INFO:teuthology.orchestra.run.smithi181.stdout: "2(1)" 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:40.726 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:40.749 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existing_9', 'key': '', 'snapid': -2, 'hash': 1217469130, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'3", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(1)']}, {'oid': {'oid': 'new_9', 'key': '', 'snapid': -2, 'hash': 170713514, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(1)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:40.749 INFO:tasks.ec_lost_unfound:reverting unfound in 1.a 2024-09-15T21:03:40.749 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.a mark_unfound_lost delete 2024-09-15T21:03:40.987 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:41.010 INFO:tasks.ec_lost_unfound:listing missing/lost in 1.b state active+recovery_unfound+undersized+degraded 2024-09-15T21:03:41.010 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 1.b list_unfound '{}' 2024-09-15T21:03:41.248 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:41.248 INFO:teuthology.orchestra.run.smithi181.stdout: "offset": { 2024-09-15T21:03:41.248 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "", 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": 0, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 0, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": -9223372036854775808, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "num_missing": 2, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "num_unfound": 2, 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "objects": [ 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:41.249 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existed_3", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 943960171, 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'5", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "0'0", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:41.250 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": { 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "oid": "existing_1", 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "key": "", 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "snapid": -2, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "hash": 2885884443, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "max": 0, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "pool": 1, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "namespace": "" 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "need": "19'4", 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "have": "16'1", 2024-09-15T21:03:41.251 INFO:teuthology.orchestra.run.smithi181.stdout: "flags": "none", 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "clean_regions": "clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false", 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "locations": [ 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "2(2)" 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "state": "NotRecovering", 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "available_might_have_unfound": true, 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "might_have_unfound": [ 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "osd": "1(0)", 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: "status": "osd is down" 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:41.252 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:41.253 INFO:teuthology.orchestra.run.smithi181.stdout: "more": false 2024-09-15T21:03:41.253 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:41.264 INFO:tasks.ec_lost_unfound:{'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 2, 'num_unfound': 2, 'objects': [{'oid': {'oid': 'existed_3', 'key': '', 'snapid': -2, 'hash': 943960171, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'5", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['2(2)']}, {'oid': {'oid': 'existing_1', 'key': '', 'snapid': -2, 'hash': 2885884443, 'max': 0, 'pool': 1, 'namespace': ''}, 'need': "19'4", 'have': "16'1", 'flags': 'none', 'clean_regions': 'clean_offsets: [(6790, 18446744073709544825)], clean_omap: true, new_object: false', 'locations': ['2(2)']}], 'state': 'NotRecovering', 'available_might_have_unfound': True, 'might_have_unfound': [{'osd': '1(0)', 'status': 'osd is down'}]} 2024-09-15T21:03:41.264 INFO:tasks.ec_lost_unfound:reverting unfound in 1.b 2024-09-15T21:03:41.264 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 1.b mark_unfound_lost delete 2024-09-15T21:03:41.536 INFO:teuthology.orchestra.run.smithi181.stderr:pg has 2 objects unfound and apparently lost marking 2024-09-15T21:03:41.560 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 5 2024-09-15T21:03:41.629 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 18 20 23036 23016 9.9879 7.82812 0.0145159 0.0156346 2024-09-15T21:03:41.788 INFO:teuthology.orchestra.run.smithi181.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T21:03:41.813 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 5 2024-09-15T21:03:42.036 INFO:teuthology.orchestra.run.smithi181.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T21:03:42.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 5 2024-09-15T21:03:42.321 INFO:teuthology.orchestra.run.smithi181.stderr:kicking recovery queue. set osd_recovery_delay_start to 5 2024-09-15T21:03:42.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T21:03:42.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T21:03:42.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T21:03:42.623 INFO:teuthology.orchestra.run.smithi181.stdout:38654705685 2024-09-15T21:03:42.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:42.629 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 19 20 23926 23906 9.82816 6.95312 0.0253177 0.0158839 2024-09-15T21:03:42.639 INFO:teuthology.orchestra.run.smithi181.stdout:111669149706 2024-09-15T21:03:42.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:42.654 INFO:teuthology.orchestra.run.smithi181.stdout:98784247820 2024-09-15T21:03:42.654 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:43.212 INFO:teuthology.orchestra.run.smithi181.stdout:38654705684 2024-09-15T21:03:43.235 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705684 for osd.2 2024-09-15T21:03:43.260 INFO:teuthology.orchestra.run.smithi181.stdout:111669149704 2024-09-15T21:03:43.261 INFO:teuthology.orchestra.run.smithi181.stdout:98784247818 2024-09-15T21:03:43.287 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149704 for osd.3 2024-09-15T21:03:43.288 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247818 for osd.0 2024-09-15T21:03:43.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:03:43.627650+0000 min lat: 0.00413303 max lat: 0.0651775 avg lat: 0.0161576 2024-09-15T21:03:43.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:03:43.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 20 20 24762 24742 9.6633 6.53125 0.0137742 0.0161576 2024-09-15T21:03:44.230 INFO:tasks.ceph.mon.c.smithi181.stderr:2024-09-15T21:03:44.227+0000 7f52db2c9640 -1 log_channel(cluster) log [ERR] : Health check update: Possible data damage: 7 pgs recovery_unfound (PG_DAMAGED) 2024-09-15T21:03:44.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T21:03:44.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T21:03:44.289 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T21:03:44.629 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 21 20 25679 25659 9.54427 7.16406 0.0153652 0.0163578 2024-09-15T21:03:44.756 INFO:teuthology.orchestra.run.smithi181.stdout:38654705685 2024-09-15T21:03:44.779 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 38654705685 got 38654705685 for osd.2 2024-09-15T21:03:44.779 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:44.941 INFO:teuthology.orchestra.run.smithi181.stdout:111669149706 2024-09-15T21:03:44.954 INFO:teuthology.orchestra.run.smithi181.stdout:98784247820 2024-09-15T21:03:44.964 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 111669149706 got 111669149706 for osd.3 2024-09-15T21:03:44.964 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:44.978 INFO:tasks.ec_lost_unfound.ceph_manager:need seq 98784247820 got 98784247820 for osd.0 2024-09-15T21:03:44.978 DEBUG:teuthology.parallel:result is None 2024-09-15T21:03:44.978 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for recovery to complete 2024-09-15T21:03:44.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:03:45.567 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:45.568 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:03:45.592 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-15T21:03:43.913417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195905192,"num_objects":23919,"num_object_clones":0,"num_object_copies":95676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":191450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23973,"ondisk_log_size":23973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":582688,"kb_used_data":282076,"kb_used_omap":11,"kb_used_meta":300532,"kb_avail":376904672,"statfs":{"total":386547056640,"available":385950384128,"internally_reserved":0,"allocated":288845824,"data_stored":288324459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11992,"internal_metadata":307745064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,2,6,9,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":136732672,"num_objects":16691,"num_object_clones":0,"num_object_copies":66764,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":16637,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":16691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16691,"num_write_kb":133528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003220"},"pg_stats":[{"pgid":"1.f","version":"27'1449","reported_seq":1478,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.555339+0000","last_change":"2024-09-15T21:03:23.636949+0000","last_active":"2024-09-15T21:03:42.555339+0000","last_peered":"2024-09-15T21:03:42.555339+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:42.555339+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:11:10.422591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11870208,"num_objects":1449,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":11592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1561","reported_seq":1609,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.614730+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:42.614730+0000","last_peered":"2024-09-15T21:03:42.614730+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:42.614730+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12752140,"num_objects":1557,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1559,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1557}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1442","reported_seq":1502,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.584972+0000","last_change":"2024-09-15T21:03:35.222367+0000","last_active":"2024-09-15T21:03:42.584972+0000","last_peered":"2024-09-15T21:03:42.584972+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:42.584972+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1442,"log_dups_size":0,"ondisk_log_size":1442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.621605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11786886,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":11518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1480","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.560042+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:42.560042+0000","last_peered":"2024-09-15T21:03:42.560042+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:42.560042+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12088588,"num_objects":1476,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1505","reported_seq":1545,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.554891+0000","last_change":"2024-09-15T21:03:36.329429+0000","last_active":"2024-09-15T21:03:42.554891+0000","last_peered":"2024-09-15T21:03:42.554891+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:42.554891+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.934838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301580,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1503,"num_write_kb":12021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1513","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.544293+0000","last_change":"2024-09-15T21:03:23.649982+0000","last_active":"2024-09-15T21:03:42.544293+0000","last_peered":"2024-09-15T21:03:42.544293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:42.544293+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1513,"log_dups_size":0,"ondisk_log_size":1513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:23.850087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12394496,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":12104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1555","reported_seq":1606,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.583792+0000","last_change":"2024-09-15T21:03:36.825186+0000","last_active":"2024-09-15T21:03:42.583792+0000","last_peered":"2024-09-15T21:03:42.583792+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:42.583792+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:37.475259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12686604,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1570","reported_seq":1621,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.558271+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:42.558271+0000","last_peered":"2024-09-15T21:03:42.558271+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:42.558271+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817676,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":12532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1540","reported_seq":1581,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.598270+0000","last_change":"2024-09-15T21:03:37.927187+0000","last_active":"2024-09-15T21:03:42.598270+0000","last_peered":"2024-09-15T21:03:42.598270+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:42.598270+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12544536,"num_objects":1532,"num_object_clones":0,"num_object_copies":6128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":12273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1496","reported_seq":1559,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.523034+0000","last_change":"2024-09-15T21:03:38.428321+0000","last_active":"2024-09-15T21:03:42.523034+0000","last_peered":"2024-09-15T21:03:42.523034+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:42.523034+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.659794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12236044,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":11950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1494}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1495","reported_seq":1537,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.542361+0000","last_change":"2024-09-15T21:03:39.013607+0000","last_active":"2024-09-15T21:03:42.542361+0000","last_peered":"2024-09-15T21:03:42.542361+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:42.542361+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:39.888243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12201874,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":11924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1465","reported_seq":1489,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.572239+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:42.572239+0000","last_peered":"2024-09-15T21:03:42.572239+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:42.572239+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11991686,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":11711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1462","reported_seq":1523,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.558898+0000","last_change":"2024-09-15T21:03:39.975839+0000","last_active":"2024-09-15T21:03:42.558898+0000","last_peered":"2024-09-15T21:03:42.558898+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:42.558898+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11967110,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1465","reported_seq":1503,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.548973+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:42.548973+0000","last_peered":"2024-09-15T21:03:42.548973+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:42.548973+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11965708,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":11700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1461","reported_seq":1526,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.573452+0000","last_change":"2024-09-15T21:03:41.067733+0000","last_active":"2024-09-15T21:03:42.573452+0000","last_peered":"2024-09-15T21:03:42.573452+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:42.573452+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1461,"log_dups_size":0,"ondisk_log_size":1461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.756240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1459,"num_write_kb":11669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1514","reported_seq":1563,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.556771+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:42.556771+0000","last_peered":"2024-09-15T21:03:42.556771+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:42.556771+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358924,"num_objects":1509,"num_object_clones":0,"num_object_copies":6036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1512,"num_write_kb":12084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195905192,"num_objects":23919,"num_object_clones":0,"num_object_copies":95676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":191450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":293740544,"data_stored":293740544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23973,"ondisk_log_size":23973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191976,"kb_used_data":93652,"kb_used_omap":4,"kb_used_meta":98299,"kb_avail":94179864,"statfs":{"total":96636764160,"available":96440180736,"internally_reserved":0,"allocated":95899648,"data_stored":95757743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":100658896},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173988,"kb_used_data":92820,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94197852,"statfs":{"total":96636764160,"available":96458600448,"internally_reserved":0,"allocated":95047680,"data_stored":94905775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":76382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189532,"kb_used_data":95436,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94182308,"statfs":{"total":96636764160,"available":96442683392,"internally_reserved":0,"allocated":97726464,"data_stored":97584559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":96333518},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97959936,"data_stored":97959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97857536,"data_stored":97857536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97882112,"data_stored":97882112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:03:45.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:03:45.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 22 20 26556 26536 9.4214 6.85156 0.0100952 0.016576 2024-09-15T21:03:46.154 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:03:46.154 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:03:46.181 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-15T21:03:45.913878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":195905192,"num_objects":23919,"num_object_clones":0,"num_object_copies":95676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":191450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23973,"ondisk_log_size":23973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":582688,"kb_used_data":282076,"kb_used_omap":11,"kb_used_meta":300532,"kb_avail":376904672,"statfs":{"total":386547056640,"available":385950384128,"internally_reserved":0,"allocated":288845824,"data_stored":288324459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11992,"internal_metadata":307745064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,2,6,9,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":92798976,"num_objects":11328,"num_object_clones":0,"num_object_copies":45312,"num_objects_missing_on_primary":-21,"num_objects_missing":-21,"num_objects_degraded":11274,"num_objects_misplaced":0,"num_objects_unfound":-27,"num_objects_dirty":11328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11328,"num_write_kb":90624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003102"},"pg_stats":[{"pgid":"1.f","version":"27'1449","reported_seq":1478,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.555339+0000","last_change":"2024-09-15T21:03:23.636949+0000","last_active":"2024-09-15T21:03:42.555339+0000","last_peered":"2024-09-15T21:03:42.555339+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:42.555339+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:11:10.422591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11870208,"num_objects":1449,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1449,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":11592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'1561","reported_seq":1609,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.614730+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:42.614730+0000","last_peered":"2024-09-15T21:03:42.614730+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:42.614730+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12752140,"num_objects":1557,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1557,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1559,"num_write_kb":12468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1557}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'1442","reported_seq":1502,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.584972+0000","last_change":"2024-09-15T21:03:35.222367+0000","last_active":"2024-09-15T21:03:42.584972+0000","last_peered":"2024-09-15T21:03:42.584972+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:42.584972+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1442,"log_dups_size":0,"ondisk_log_size":1442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.621605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11786886,"num_objects":1439,"num_object_clones":0,"num_object_copies":5756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1439,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1441,"num_write_kb":11518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1439}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'1480","reported_seq":1530,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.560042+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:42.560042+0000","last_peered":"2024-09-15T21:03:42.560042+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:42.560042+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12088588,"num_objects":1476,"num_object_clones":0,"num_object_copies":5904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1478,"num_write_kb":11813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1476}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"27'1505","reported_seq":1545,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.554891+0000","last_change":"2024-09-15T21:03:36.329429+0000","last_active":"2024-09-15T21:03:42.554891+0000","last_peered":"2024-09-15T21:03:42.554891+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:42.554891+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.934838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12301580,"num_objects":1502,"num_object_clones":0,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1502,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1503,"num_write_kb":12021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1502}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"27'1513","reported_seq":1542,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.544293+0000","last_change":"2024-09-15T21:03:23.649982+0000","last_active":"2024-09-15T21:03:42.544293+0000","last_peered":"2024-09-15T21:03:42.544293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:42.544293+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1513,"log_dups_size":0,"ondisk_log_size":1513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:17:23.850087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12394496,"num_objects":1513,"num_object_clones":0,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1513,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1513,"num_write_kb":12104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"27'1555","reported_seq":1606,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.583792+0000","last_change":"2024-09-15T21:03:36.825186+0000","last_active":"2024-09-15T21:03:42.583792+0000","last_peered":"2024-09-15T21:03:42.583792+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:42.583792+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:37.475259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12686604,"num_objects":1549,"num_object_clones":0,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":12404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2147483647,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1549}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'1570","reported_seq":1621,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.558271+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:42.558271+0000","last_peered":"2024-09-15T21:03:42.558271+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:42.558271+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12817676,"num_objects":1565,"num_object_clones":0,"num_object_copies":6260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1565,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1568,"num_write_kb":12532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"27'1540","reported_seq":1581,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.598270+0000","last_change":"2024-09-15T21:03:37.927187+0000","last_active":"2024-09-15T21:03:42.598270+0000","last_peered":"2024-09-15T21:03:42.598270+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:42.598270+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.911953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12544536,"num_objects":1532,"num_object_clones":0,"num_object_copies":6128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1537,"num_write_kb":12273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,2147483647],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'1496","reported_seq":1559,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.523034+0000","last_change":"2024-09-15T21:03:38.428321+0000","last_active":"2024-09-15T21:03:42.523034+0000","last_peered":"2024-09-15T21:03:42.523034+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:42.523034+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1496,"log_dups_size":0,"ondisk_log_size":1496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.659794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12236044,"num_objects":1494,"num_object_clones":0,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1494,"num_write_kb":11950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1494}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'1495","reported_seq":1537,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.542361+0000","last_change":"2024-09-15T21:03:39.013607+0000","last_active":"2024-09-15T21:03:42.542361+0000","last_peered":"2024-09-15T21:03:42.542361+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:42.542361+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:39.888243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12201874,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1492,"num_write_kb":11924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2147483647,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1465","reported_seq":1489,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.572239+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:42.572239+0000","last_peered":"2024-09-15T21:03:42.572239+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:42.572239+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11991686,"num_objects":1464,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1464,"num_write_kb":11711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'1462","reported_seq":1523,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.558898+0000","last_change":"2024-09-15T21:03:39.975839+0000","last_active":"2024-09-15T21:03:42.558898+0000","last_peered":"2024-09-15T21:03:42.558898+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:42.558898+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1462,"log_dups_size":0,"ondisk_log_size":1462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11967110,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1461,"num_write_kb":11687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'1465","reported_seq":1503,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.548973+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:42.548973+0000","last_peered":"2024-09-15T21:03:42.548973+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:42.548973+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11965708,"num_objects":1461,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":11700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'1461","reported_seq":1526,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.573452+0000","last_change":"2024-09-15T21:03:41.067733+0000","last_active":"2024-09-15T21:03:42.573452+0000","last_peered":"2024-09-15T21:03:42.573452+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:42.573452+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1461,"log_dups_size":0,"ondisk_log_size":1461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.756240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11941132,"num_objects":1458,"num_object_clones":0,"num_object_copies":5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1459,"num_write_kb":11669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1458}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'1514","reported_seq":1563,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:42.556771+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:42.556771+0000","last_peered":"2024-09-15T21:03:42.556771+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:42.556771+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12358924,"num_objects":1509,"num_object_clones":0,"num_object_copies":6036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1512,"num_write_kb":12084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1509}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":195905192,"num_objects":23919,"num_object_clones":0,"num_object_copies":95676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23919,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":191450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":293740544,"data_stored":293740544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23973,"ondisk_log_size":23973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191976,"kb_used_data":93652,"kb_used_omap":4,"kb_used_meta":98299,"kb_avail":94179864,"statfs":{"total":96636764160,"available":96440180736,"internally_reserved":0,"allocated":95899648,"data_stored":95757743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":100658896},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":173988,"kb_used_data":92820,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":94197852,"statfs":{"total":96636764160,"available":96458600448,"internally_reserved":0,"allocated":95047680,"data_stored":94905775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":172032,"data_stored":76382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":189532,"kb_used_data":95436,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94182308,"statfs":{"total":96636764160,"available":96442683392,"internally_reserved":0,"allocated":97726464,"data_stored":97584559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":96333518},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97959936,"data_stored":97959936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97857536,"data_stored":97857536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":97882112,"data_stored":97882112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:03:46.182 INFO:tasks.ec_lost_unfound.ceph_manager:recovered! 2024-09-15T21:03:46.182 INFO:tasks.util.rados:rados -p unique_pool_0 get new_1 - 2024-09-15T21:03:46.182 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_1 - 2024-09-15T21:03:46.325 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_1: (2) No such file or directory 2024-09-15T21:03:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.333 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_1 - 2024-09-15T21:03:46.333 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_1 - 2024-09-15T21:03:46.427 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_1: (2) No such file or directory 2024-09-15T21:03:46.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.438 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_1 - 2024-09-15T21:03:46.438 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_1 - 2024-09-15T21:03:46.561 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_1: (2) No such file or directory 2024-09-15T21:03:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.569 INFO:tasks.util.rados:rados -p unique_pool_0 get new_2 - 2024-09-15T21:03:46.570 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_2 - 2024-09-15T21:03:46.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 23 20 27482 27462 9.3263 7.23438 0.0174341 0.0167461 2024-09-15T21:03:46.662 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_2: (2) No such file or directory 2024-09-15T21:03:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.673 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_2 - 2024-09-15T21:03:46.673 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_2 - 2024-09-15T21:03:46.767 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_2: (2) No such file or directory 2024-09-15T21:03:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.774 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_2 - 2024-09-15T21:03:46.774 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_2 - 2024-09-15T21:03:46.873 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_2: (2) No such file or directory 2024-09-15T21:03:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.879 INFO:tasks.util.rados:rados -p unique_pool_0 get new_3 - 2024-09-15T21:03:46.879 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_3 - 2024-09-15T21:03:46.983 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_3: (2) No such file or directory 2024-09-15T21:03:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:46.990 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_3 - 2024-09-15T21:03:46.990 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_3 - 2024-09-15T21:03:47.099 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_3: (2) No such file or directory 2024-09-15T21:03:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.105 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_3 - 2024-09-15T21:03:47.105 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_3 - 2024-09-15T21:03:47.206 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_3: (2) No such file or directory 2024-09-15T21:03:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.212 INFO:tasks.util.rados:rados -p unique_pool_0 get new_4 - 2024-09-15T21:03:47.212 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_4 - 2024-09-15T21:03:47.326 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_4: (2) No such file or directory 2024-09-15T21:03:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.336 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_4 - 2024-09-15T21:03:47.336 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_4 - 2024-09-15T21:03:47.437 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_4: (2) No such file or directory 2024-09-15T21:03:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.449 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_4 - 2024-09-15T21:03:47.449 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_4 - 2024-09-15T21:03:47.555 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_4: (2) No such file or directory 2024-09-15T21:03:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.561 INFO:tasks.util.rados:rados -p unique_pool_0 get new_5 - 2024-09-15T21:03:47.561 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_5 - 2024-09-15T21:03:47.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 24 20 28379 28359 9.22968 7.00781 0.0174606 0.0169186 2024-09-15T21:03:47.661 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_5: (2) No such file or directory 2024-09-15T21:03:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.670 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_5 - 2024-09-15T21:03:47.670 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_5 - 2024-09-15T21:03:47.761 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_5: (2) No such file or directory 2024-09-15T21:03:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.770 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_5 - 2024-09-15T21:03:47.770 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_5 - 2024-09-15T21:03:47.874 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_5: (2) No such file or directory 2024-09-15T21:03:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:47.885 INFO:tasks.util.rados:rados -p unique_pool_0 get new_6 - 2024-09-15T21:03:47.885 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_6 - 2024-09-15T21:03:47.998 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_6: (2) No such file or directory 2024-09-15T21:03:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.001 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_6 - 2024-09-15T21:03:48.002 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_6 - 2024-09-15T21:03:48.096 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_6: (2) No such file or directory 2024-09-15T21:03:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.109 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_6 - 2024-09-15T21:03:48.109 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_6 - 2024-09-15T21:03:48.211 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_6: (2) No such file or directory 2024-09-15T21:03:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.217 INFO:tasks.util.rados:rados -p unique_pool_0 get new_7 - 2024-09-15T21:03:48.218 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_7 - 2024-09-15T21:03:48.330 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_7: (2) No such file or directory 2024-09-15T21:03:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.342 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_7 - 2024-09-15T21:03:48.343 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_7 - 2024-09-15T21:03:48.459 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_7: (2) No such file or directory 2024-09-15T21:03:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.470 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_7 - 2024-09-15T21:03:48.470 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_7 - 2024-09-15T21:03:48.568 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_7: (2) No such file or directory 2024-09-15T21:03:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.572 INFO:tasks.util.rados:rados -p unique_pool_0 get new_8 - 2024-09-15T21:03:48.573 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_8 - 2024-09-15T21:03:48.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 25 20 29287 29267 9.14423 7.09375 0.0153668 0.0170735 2024-09-15T21:03:48.672 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_8: (2) No such file or directory 2024-09-15T21:03:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.681 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_8 - 2024-09-15T21:03:48.681 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_8 - 2024-09-15T21:03:48.786 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_8: (2) No such file or directory 2024-09-15T21:03:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.801 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_8 - 2024-09-15T21:03:48.801 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_8 - 2024-09-15T21:03:48.925 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_8: (2) No such file or directory 2024-09-15T21:03:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:48.931 INFO:tasks.util.rados:rados -p unique_pool_0 get new_9 - 2024-09-15T21:03:48.931 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get new_9 - 2024-09-15T21:03:49.021 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/new_9: (2) No such file or directory 2024-09-15T21:03:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:49.035 INFO:tasks.util.rados:rados -p unique_pool_0 get existed_9 - 2024-09-15T21:03:49.035 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existed_9 - 2024-09-15T21:03:49.150 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existed_9: (2) No such file or directory 2024-09-15T21:03:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:49.156 INFO:tasks.util.rados:rados -p unique_pool_0 get existing_9 - 2024-09-15T21:03:49.156 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 get existing_9 - 2024-09-15T21:03:49.256 INFO:teuthology.orchestra.run.smithi181.stderr:error getting unique_pool_0/existing_9: (2) No such file or directory 2024-09-15T21:03:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:03:49.263 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T21:03:49.263 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T21:03:49.266 INFO:tasks.ceph.osd.1:Started 2024-09-15T21:03:49.267 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:03:49.447 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-15T21:03:49.448 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:03:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:03:49.450 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T21:03:49.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 26 20 30191 30171 9.06415 7.0625 0.0242697 0.0172279 2024-09-15T21:03:50.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 27 20 31187 31167 9.01662 7.78125 0.0119989 0.0173196 2024-09-15T21:03:50.730 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:03:50.724+0000 7f764f818640 -1 Falling back to public interface 2024-09-15T21:03:51.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 28 20 32150 32130 8.96328 7.52344 0.0199127 0.0174209 2024-09-15T21:03:52.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 29 20 33068 33048 8.90149 7.17188 0.0659286 0.0175338 2024-09-15T21:03:53.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 30 20 34032 34012 8.8558 7.53125 0.0147098 0.0176357 2024-09-15T21:03:54.452 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:no valid command found; 10 closest matches: 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:0 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:1 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:2 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:abort 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:assert 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:config diff 2024-09-15T21:03:54.627 INFO:teuthology.orchestra.run.smithi181.stderr:config diff get 2024-09-15T21:03:54.628 INFO:teuthology.orchestra.run.smithi181.stderr:config get 2024-09-15T21:03:54.628 INFO:teuthology.orchestra.run.smithi181.stderr:config help [] 2024-09-15T21:03:54.628 INFO:teuthology.orchestra.run.smithi181.stderr:config set ... 2024-09-15T21:03:54.628 INFO:teuthology.orchestra.run.smithi181.stderr:admin_socket: invalid command 2024-09-15T21:03:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-15T21:03:54.630 INFO:tasks.ec_lost_unfound.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-09-15T21:03:54.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 31 20 34900 34880 8.78887 6.78125 0.0242264 0.0177652 2024-09-15T21:03:55.379 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:03:55.375+0000 7f764f818640 -1 osd.1 19 log_to_monitors true 2024-09-15T21:03:55.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 32 20 35798 35778 8.73344 7.01562 0.0174915 0.0178813 2024-09-15T21:03:56.038 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:03:56.034+0000 7f7643043640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T21:03:56.631 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 33 20 36707 36687 8.68398 7.10156 0.0223358 0.0179847 2024-09-15T21:03:57.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 34 20 37064 37044 8.51059 2.78906 0.0333489 0.0180279 2024-09-15T21:03:58.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 35 20 37064 37044 8.26743 0 - 0.0180279 2024-09-15T21:03:59.631 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T21:03:59.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 36 20 37067 37047 8.03845 0.0117188 0.0098499 0.0181669 2024-09-15T21:03:59.806 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "ops": [ 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.as0 31/30 ECSubWrite(tid=12971, reqid=client.5334.0:37099, at_version=31'2289, trim_to=0'0, roll_forward_to=31'2287))", 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.797037+0000", 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.006986595, 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0070124690000000003, 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.807 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797037+0000", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797037+0000", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797040+0000", 2024-09-15T21:03:59.808 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5449999999999999e-06 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797070+0000", 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.9280000000000001e-05 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797080+0000", 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0091e-05 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.809 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797088+0000", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.0420000000000003e-06 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797675+0000", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.00058704600000000003 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797684+0000", 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.9129999999999992e-06 2024-09-15T21:03:59.810 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_committed", 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.803983+0000", 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0062991619999999996 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.5s3 31/30 ECSubWrite(tid=12972, reqid=client.5334.0:37101, at_version=31'2340, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.797522+0000", 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0065017870000000002, 2024-09-15T21:03:59.811 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065875300000000003, 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797522+0000", 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797522+0000", 2024-09-15T21:03:59.812 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797522+0000", 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.3799999999999998e-07 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797547+0000", 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.4374000000000001e-05 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.813 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797556+0000", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.7629999999999995e-06 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797564+0000", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3459999999999996e-06 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799685+0000", 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0021213429999999999 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.814 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799694+0000", 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 9.0880000000000001e-06 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.ds1 31/30 ECSubWrite(tid=12973, reqid=client.5334.0:37105, at_version=31'2265, trim_to=0'0, roll_forward_to=31'2264))", 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.797581+0000", 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0064428389999999997, 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0065778649999999996, 2024-09-15T21:03:59.815 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797581+0000", 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797581+0000", 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.816 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797581+0000", 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.5400000000000002e-07 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797596+0000", 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.4645e-05 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.817 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797603+0000", 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.0679999999999999e-06 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.797608+0000", 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.4199999999999998e-06 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.818 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800529+0000", 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002920579 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800537+0000", 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2290000000000003e-06 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.819 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.4s3 31/30 ECSubWrite(tid=22690, reqid=client.5334.0:37083, at_version=0'0, trim_to=0'0, roll_forward_to=31'2346))", 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.799445+0000", 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0045785089999999997, 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0047591059999999999, 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799445+0000", 2024-09-15T21:03:59.820 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799445+0000", 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.821 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799446+0000", 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9800000000000003e-07 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799458+0000", 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.2002e-05 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799469+0000", 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0856000000000001e-05 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.822 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.799476+0000", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.3989999999999998e-06 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.801780+0000", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.002303724 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.801788+0000", 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.2300000000000008e-06 2024-09-15T21:03:59.823 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.es0 31/30 ECSubWrite(tid=16507, reqid=client.5334.0:37077, at_version=0'0, trim_to=0'0, roll_forward_to=31'2390))", 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.800837+0000", 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0031862489999999999, 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0034143009999999998, 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.824 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800837+0000", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800837+0000", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800838+0000", 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.23e-07 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.825 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800852+0000", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.3577999999999999e-05 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800862+0000", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 1.0791000000000001e-05 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800870+0000", 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.7000000000000008e-06 2024-09-15T21:03:59.826 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.801017+0000", 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.000146568 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.801025+0000", 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 8.8680000000000005e-06 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.827 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "description": "MOSDECSubOpWrite(1.0s0 31/30 ECSubWrite(tid=16508, reqid=client.5334.0:37111, at_version=31'2341, trim_to=0'0, roll_forward_to=0'0))", 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "initiated_at": "2024-09-15T21:03:59.800901+0000", 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "age": 0.0031224999999999998, 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0.0033949589999999999, 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "continuous": false, 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "type_data": { 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "flag_point": "reached pg", 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "events": [ 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "initiated", 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800901+0000", 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.828 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "header_read", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800901+0000", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 0 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "throttled", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800901+0000", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 2.84e-07 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "all_read", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800931+0000", 2024-09-15T21:03:59.829 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 3.0151e-05 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "dispatched", 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800939+0000", 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9629999999999998e-06 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "queued_for_pg", 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800945+0000", 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 5.6509999999999998e-06 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "reached_pg", 2024-09-15T21:03:59.830 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800991+0000", 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 4.5852000000000001e-05 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: }, 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: { 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: "event": "sub_op_started", 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: "time": "2024-09-15T21:03:59.800999+0000", 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: "duration": 7.9689999999999994e-06 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: } 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: ], 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout: "num_ops": 6 2024-09-15T21:03:59.831 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:03:59.832 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for osd.1 to be up 2024-09-15T21:03:59.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:04:00.435 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:00.435 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":31,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:03:59.037132+0000","last_up_change":"2024-09-15T21:03:57.008004+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T21:02:37.977251+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":30,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6809","nonce":2996802086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6811","nonce":2996802086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6815","nonce":2996802086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6813","nonce":2996802086}]},"public_addr":"172.21.15.181:6809/2996802086","cluster_addr":"172.21.15.181:6811/2996802086","heartbeat_back_addr":"172.21.15.181:6815/2996802086","heartbeat_front_addr":"172.21.15.181:6813/2996802086","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":29,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6825","nonce":36577163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6827","nonce":36577163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6831","nonce":36577163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6829","nonce":36577163}]},"public_addr":"172.21.15.181:6825/36577163","cluster_addr":"172.21.15.181:6827/36577163","heartbeat_back_addr":"172.21.15.181:6831/36577163","heartbeat_front_addr":"172.21.15.181:6829/36577163","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":30,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6817","nonce":2345571741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6819","nonce":2345571741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6823","nonce":2345571741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6821","nonce":2345571741}]},"public_addr":"172.21.15.181:6817/2345571741","cluster_addr":"172.21.15.181:6819/2345571741","heartbeat_back_addr":"172.21.15.181:6823/2345571741","heartbeat_front_addr":"172.21.15.181:6821/2345571741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[2147483647,0,2,3]},{"pgid":"1.1","osds":[3,2147483647,0,2]},{"pgid":"1.2","osds":[0,3,2147483647,2]},{"pgid":"1.3","osds":[2147483647,0,2,3]},{"pgid":"1.4","osds":[3,0,2,2147483647]},{"pgid":"1.5","osds":[2,0,3,2147483647]},{"pgid":"1.6","osds":[3,0,2147483647,2]},{"pgid":"1.7","osds":[2147483647,3,2,0]},{"pgid":"1.8","osds":[2147483647,2,3,0]},{"pgid":"1.9","osds":[2147483647,3,2,0]},{"pgid":"1.a","osds":[2147483647,2,3,0]},{"pgid":"1.b","osds":[2147483647,0,2,3]},{"pgid":"1.c","osds":[3,0,2,2147483647]},{"pgid":"1.d","osds":[2,2147483647,3,0]},{"pgid":"1.e","osds":[2147483647,0,3,2]},{"pgid":"1.f","osds":[3,0,2147483647,2]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:04:00.464 INFO:tasks.ec_lost_unfound.ceph_manager:osd.1 is up 2024-09-15T21:04:00.464 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T21:04:00.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:00.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 37 20 37417 37397 7.8951 2.73438 0.0470957 0.0197703 2024-09-15T21:04:01.067 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:01.067 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:01.105 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-15T21:03:59.917002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":288466600,"num_objects":35218,"num_object_clones":0,"num_object_copies":140872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35245,"num_write_kb":281842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35272,"ondisk_log_size":35272,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864188,"kb_used_data":439540,"kb_used_omap":14,"kb_used_meta":424561,"kb_avail":376623172,"statfs":{"total":386547056640,"available":385662128128,"internally_reserved":0,"allocated":450088960,"data_stored":449547020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":434751026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,9,11,9,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-11130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000869"},"pg_stats":[{"pgid":"1.f","version":"28'2276","reported_seq":2307,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.036836+0000","last_change":"2024-09-15T21:03:57.049281+0000","last_active":"2024-09-15T21:03:57.049281+0000","last_peered":"2024-09-15T21:03:57.036836+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:57.036836+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18644992,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2276,"num_write_kb":18208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'2113","reported_seq":2164,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.251947+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:52.251947+0000","last_peered":"2024-09-15T21:03:52.251947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:52.251947+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274124,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'2202","reported_seq":2263,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.966658+0000","last_change":"2024-09-15T21:03:35.222367+0000","last_active":"2024-09-15T21:03:55.966658+0000","last_peered":"2024-09-15T21:03:55.966658+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:55.966658+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.621605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18012806,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'2064","reported_seq":2115,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.231179+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:52.231179+0000","last_peered":"2024-09-15T21:03:52.231179+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:52.231179+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16872716,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":16485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2060}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2316","reported_seq":2356,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.995686+0000","last_change":"2024-09-15T21:03:57.047159+0000","last_active":"2024-09-15T21:03:57.047159+0000","last_peered":"2024-09-15T21:03:56.995686+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:56.995686+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":18509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2316","reported_seq":2346,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.981287+0000","last_change":"2024-09-15T21:03:57.047525+0000","last_active":"2024-09-15T21:03:57.047525+0000","last_peered":"2024-09-15T21:03:56.981287+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:56.981287+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":18528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2374","reported_seq":2428,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.017515+0000","last_change":"2024-09-15T21:03:57.043225+0000","last_active":"2024-09-15T21:03:57.043225+0000","last_peered":"2024-09-15T21:03:57.017515+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:57.017515+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19395852,"num_objects":2368,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":18956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'2173","reported_seq":2223,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.235230+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:52.235230+0000","last_peered":"2024-09-15T21:03:52.235230+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:52.235230+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2344","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.016784+0000","last_change":"2024-09-15T21:03:57.046439+0000","last_active":"2024-09-15T21:03:57.046439+0000","last_peered":"2024-09-15T21:03:57.016784+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:57.016784+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19130904,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":18705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'2279","reported_seq":2342,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.945109+0000","last_change":"2024-09-15T21:03:38.428321+0000","last_active":"2024-09-15T21:03:55.945109+0000","last_peered":"2024-09-15T21:03:55.945109+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:55.945109+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.659794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18650380,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":18214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2268","reported_seq":2312,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.999012+0000","last_change":"2024-09-15T21:03:57.044257+0000","last_active":"2024-09-15T21:03:57.044257+0000","last_peered":"2024-09-15T21:03:56.999012+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:56.999012+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18534290,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2265,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1994","reported_seq":2018,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.313143+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:52.313143+0000","last_peered":"2024-09-15T21:03:52.313143+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:52.313143+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1994,"log_dups_size":0,"ondisk_log_size":1994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16325254,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2234","reported_seq":2295,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.932513+0000","last_change":"2024-09-15T21:03:39.975839+0000","last_active":"2024-09-15T21:03:55.932513+0000","last_peered":"2024-09-15T21:03:55.932513+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:55.932513+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18291334,"num_objects":2233,"num_object_clones":0,"num_object_copies":8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":17863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'2028","reported_seq":2067,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.315969+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:52.315969+0000","last_peered":"2024-09-15T21:03:52.315969+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:52.315969+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577804,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2224","reported_seq":2289,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.964958+0000","last_change":"2024-09-15T21:03:41.067733+0000","last_active":"2024-09-15T21:03:55.964958+0000","last_peered":"2024-09-15T21:03:55.964958+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:55.964958+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.756240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18191628,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2221}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'2067","reported_seq":2118,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.209930+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:52.209930+0000","last_peered":"2024-09-15T21:03:52.209930+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:52.209930+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16889100,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":288466600,"num_objects":35218,"num_object_clones":0,"num_object_copies":140872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35245,"num_write_kb":281842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":451633152,"data_stored":451633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35272,"ondisk_log_size":35272,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288060,"kb_used_data":148072,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94083780,"statfs":{"total":96636764160,"available":96341790720,"internally_reserved":0,"allocated":151625728,"data_stored":151480011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265480,"kb_used_data":144056,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106360,"statfs":{"total":96636764160,"available":96364912640,"internally_reserved":0,"allocated":147513344,"data_stored":147371439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":278528,"data_stored":169927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":282904,"kb_used_data":147140,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94088936,"statfs":{"total":96636764160,"available":96347070464,"internally_reserved":0,"allocated":150671360,"data_stored":150525643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147922944,"data_stored":147922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:01.106 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:01.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 38 20 37799 37779 7.76587 2.98438 0.0183453 0.0201003 2024-09-15T21:04:01.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:01.711 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:01.747 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-15T21:03:59.917002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":288466600,"num_objects":35218,"num_object_clones":0,"num_object_copies":140872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35245,"num_write_kb":281842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35272,"ondisk_log_size":35272,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":864188,"kb_used_data":439540,"kb_used_omap":14,"kb_used_meta":424561,"kb_avail":376623172,"statfs":{"total":386547056640,"available":385662128128,"internally_reserved":0,"allocated":450088960,"data_stored":449547020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":434751026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,6,9,11,9,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-11130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2742,"num_write_kb":21936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000869"},"pg_stats":[{"pgid":"1.f","version":"28'2276","reported_seq":2307,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.036836+0000","last_change":"2024-09-15T21:03:57.049281+0000","last_active":"2024-09-15T21:03:57.049281+0000","last_peered":"2024-09-15T21:03:57.036836+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:57.036836+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18644992,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2276,"num_write_kb":18208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'2113","reported_seq":2164,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.251947+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:52.251947+0000","last_peered":"2024-09-15T21:03:52.251947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:52.251947+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274124,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"27'2202","reported_seq":2263,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.966658+0000","last_change":"2024-09-15T21:03:35.222367+0000","last_active":"2024-09-15T21:03:55.966658+0000","last_peered":"2024-09-15T21:03:55.966658+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:14.815981+0000","last_became_peered":"2024-09-15T21:03:14.815981+0000","last_unstale":"2024-09-15T21:03:55.966658+0000","last_undegraded":"2024-09-15T21:03:14.803037+0000","last_fullsized":"2024-09-15T21:03:14.802658+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2202,"log_dups_size":0,"ondisk_log_size":2202,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.621605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18012806,"num_objects":2199,"num_object_clones":0,"num_object_copies":8796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2199,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":17598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,2147483647,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'2064","reported_seq":2115,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.231179+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:52.231179+0000","last_peered":"2024-09-15T21:03:52.231179+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:52.231179+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16872716,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":16485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2060}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2316","reported_seq":2356,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.995686+0000","last_change":"2024-09-15T21:03:57.047159+0000","last_active":"2024-09-15T21:03:57.047159+0000","last_peered":"2024-09-15T21:03:56.995686+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:56.995686+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":18509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2316","reported_seq":2346,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.981287+0000","last_change":"2024-09-15T21:03:57.047525+0000","last_active":"2024-09-15T21:03:57.047525+0000","last_peered":"2024-09-15T21:03:56.981287+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:56.981287+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":18528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2374","reported_seq":2428,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.017515+0000","last_change":"2024-09-15T21:03:57.043225+0000","last_active":"2024-09-15T21:03:57.043225+0000","last_peered":"2024-09-15T21:03:57.017515+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:57.017515+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19395852,"num_objects":2368,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":18956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'2173","reported_seq":2223,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.235230+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:52.235230+0000","last_peered":"2024-09-15T21:03:52.235230+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:52.235230+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2344","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.016784+0000","last_change":"2024-09-15T21:03:57.046439+0000","last_active":"2024-09-15T21:03:57.046439+0000","last_peered":"2024-09-15T21:03:57.016784+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:57.016784+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19130904,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":18705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"27'2279","reported_seq":2342,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.945109+0000","last_change":"2024-09-15T21:03:38.428321+0000","last_active":"2024-09-15T21:03:55.945109+0000","last_peered":"2024-09-15T21:03:55.945109+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:14.815687+0000","last_became_peered":"2024-09-15T21:03:14.815687+0000","last_unstale":"2024-09-15T21:03:55.945109+0000","last_undegraded":"2024-09-15T21:03:14.803267+0000","last_fullsized":"2024-09-15T21:03:14.802940+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.659794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18650380,"num_objects":2277,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":18214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,2147483647],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2268","reported_seq":2312,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.999012+0000","last_change":"2024-09-15T21:03:57.044257+0000","last_active":"2024-09-15T21:03:57.044257+0000","last_peered":"2024-09-15T21:03:56.999012+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:56.999012+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18534290,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2265,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1994","reported_seq":2018,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.313143+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:52.313143+0000","last_peered":"2024-09-15T21:03:52.313143+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:52.313143+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1994,"log_dups_size":0,"ondisk_log_size":1994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16325254,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"27'2234","reported_seq":2295,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.932513+0000","last_change":"2024-09-15T21:03:39.975839+0000","last_active":"2024-09-15T21:03:55.932513+0000","last_peered":"2024-09-15T21:03:55.932513+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:14.811749+0000","last_became_peered":"2024-09-15T21:03:14.811749+0000","last_unstale":"2024-09-15T21:03:55.932513+0000","last_undegraded":"2024-09-15T21:03:14.805966+0000","last_fullsized":"2024-09-15T21:03:14.802444+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18291334,"num_objects":2233,"num_object_clones":0,"num_object_copies":8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":17863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2233}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'2028","reported_seq":2067,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.315969+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:52.315969+0000","last_peered":"2024-09-15T21:03:52.315969+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:52.315969+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577804,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"27'2224","reported_seq":2289,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:55.964958+0000","last_change":"2024-09-15T21:03:41.067733+0000","last_active":"2024-09-15T21:03:55.964958+0000","last_peered":"2024-09-15T21:03:55.964958+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:14.815786+0000","last_became_peered":"2024-09-15T21:03:14.815786+0000","last_unstale":"2024-09-15T21:03:55.964958+0000","last_undegraded":"2024-09-15T21:03:14.802779+0000","last_fullsized":"2024-09-15T21:03:14.802381+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2224,"log_dups_size":0,"ondisk_log_size":2224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.756240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18191628,"num_objects":2221,"num_object_clones":0,"num_object_copies":8884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":17773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2221}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'2067","reported_seq":2118,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.209930+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:52.209930+0000","last_peered":"2024-09-15T21:03:52.209930+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:52.209930+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16889100,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":288466600,"num_objects":35218,"num_object_clones":0,"num_object_copies":140872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21346,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35245,"num_write_kb":281842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":451633152,"data_stored":451633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35272,"ondisk_log_size":35272,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288060,"kb_used_data":148072,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94083780,"statfs":{"total":96636764160,"available":96341790720,"internally_reserved":0,"allocated":151625728,"data_stored":151480011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265480,"kb_used_data":144056,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":94106360,"statfs":{"total":96636764160,"available":96364912640,"internally_reserved":0,"allocated":147513344,"data_stored":147371439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":0,"seq":1,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":272,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":278528,"data_stored":169927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":28110543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":282904,"kb_used_data":147140,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94088936,"statfs":{"total":96636764160,"available":96347070464,"internally_reserved":0,"allocated":150671360,"data_stored":150525643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":114688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":147922944,"data_stored":147922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:01.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:02.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:02.326 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:02.354 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-15T21:04:01.917389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291292840,"num_objects":35563,"num_object_clones":0,"num_object_copies":142252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35590,"num_write_kb":284602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35617,"ondisk_log_size":35617,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915752,"kb_used_data":446684,"kb_used_omap":14,"kb_used_meta":468977,"kb_avail":376571608,"statfs":{"total":386547056640,"available":385609326592,"internally_reserved":0,"allocated":457404416,"data_stored":456799082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":480233010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,8,12,13],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15302656,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-12113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.906517"},"pg_stats":[{"pgid":"1.f","version":"28'2276","reported_seq":2307,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.036836+0000","last_change":"2024-09-15T21:03:57.049281+0000","last_active":"2024-09-15T21:03:57.049281+0000","last_peered":"2024-09-15T21:03:57.036836+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:57.036836+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18644992,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2276,"num_write_kb":18208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'2113","reported_seq":2164,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.251947+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:52.251947+0000","last_peered":"2024-09-15T21:03:52.251947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:52.251947+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274124,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2290","reported_seq":2363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.911607+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:00.911607+0000","last_peered":"2024-09-15T21:04:00.911607+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:00.911607+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2290,"log_dups_size":0,"ondisk_log_size":2290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18733702,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'2064","reported_seq":2115,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.231179+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:52.231179+0000","last_peered":"2024-09-15T21:03:52.231179+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:52.231179+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16872716,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":16485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2060}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2316","reported_seq":2356,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.995686+0000","last_change":"2024-09-15T21:03:57.047159+0000","last_active":"2024-09-15T21:03:57.047159+0000","last_peered":"2024-09-15T21:03:56.995686+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:56.995686+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":18509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2316","reported_seq":2346,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.981287+0000","last_change":"2024-09-15T21:03:57.047525+0000","last_active":"2024-09-15T21:03:57.047525+0000","last_peered":"2024-09-15T21:03:56.981287+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:56.981287+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":18528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2374","reported_seq":2428,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.017515+0000","last_change":"2024-09-15T21:03:57.043225+0000","last_active":"2024-09-15T21:03:57.043225+0000","last_peered":"2024-09-15T21:03:57.017515+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:57.017515+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19395852,"num_objects":2368,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":18956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'2173","reported_seq":2223,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.235230+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:52.235230+0000","last_peered":"2024-09-15T21:03:52.235230+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:52.235230+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2344","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.016784+0000","last_change":"2024-09-15T21:03:57.046439+0000","last_active":"2024-09-15T21:03:57.046439+0000","last_peered":"2024-09-15T21:03:57.016784+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:57.016784+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19130904,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":18705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2364","reported_seq":2439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.777762+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:00.777762+0000","last_peered":"2024-09-15T21:04:00.777762+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:00.777762+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19346700,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2268","reported_seq":2312,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.999012+0000","last_change":"2024-09-15T21:03:57.044257+0000","last_active":"2024-09-15T21:03:57.044257+0000","last_peered":"2024-09-15T21:03:56.999012+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:56.999012+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18534290,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2265,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1994","reported_seq":2018,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.313143+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:52.313143+0000","last_peered":"2024-09-15T21:03:52.313143+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:52.313143+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1994,"log_dups_size":0,"ondisk_log_size":1994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16325254,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2318","reported_seq":2392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.896499+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:00.896499+0000","last_peered":"2024-09-15T21:04:00.896499+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:00.896499+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2318,"log_dups_size":0,"ondisk_log_size":2318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18979462,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":18535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'2028","reported_seq":2067,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.315969+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:52.315969+0000","last_peered":"2024-09-15T21:03:52.315969+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:52.315969+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577804,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2312","reported_seq":2389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.801044+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:00.801044+0000","last_peered":"2024-09-15T21:04:00.801044+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:00.801044+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912524,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":18477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'2067","reported_seq":2118,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.209930+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:52.209930+0000","last_peered":"2024-09-15T21:03:52.209930+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:52.209930+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16889100,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291292840,"num_objects":35563,"num_object_clones":0,"num_object_copies":142252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35590,"num_write_kb":284602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459636736,"data_stored":459636736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35617,"ondisk_log_size":35617,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288060,"kb_used_data":148072,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94083780,"statfs":{"total":96636764160,"available":96341790720,"internally_reserved":0,"allocated":151625728,"data_stored":151480011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293732,"kb_used_data":150100,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94078108,"statfs":{"total":96636764160,"available":96335982592,"internally_reserved":0,"allocated":153702400,"data_stored":153545450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51056,"kb_used_data":1372,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94320784,"statfs":{"total":96636764160,"available":96584482816,"internally_reserved":0,"allocated":1404928,"data_stored":1247978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":282904,"kb_used_data":147140,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94088936,"statfs":{"total":96636764160,"available":96347070464,"internally_reserved":0,"allocated":150671360,"data_stored":150525643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2236416,"data_stored":2236416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153804800,"data_stored":153804800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "28'2276", 'reported_seq': 2307, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:57.036836+0000', 'last_change': '2024-09-15T21:03:57.049281+0000', 'last_active': '2024-09-15T21:03:57.049281+0000', 'last_peered': '2024-09-15T21:03:57.036836+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:14.814722+0000', 'last_became_peered': '2024-09-15T21:03:14.814722+0000', 'last_unstale': '2024-09-15T21:03:57.036836+0000', 'last_undegraded': '2024-09-15T21:03:23.636203+0000', 'last_fullsized': '2024-09-15T21:03:14.810067+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2276, 'log_dups_size': 0, 'ondisk_log_size': 2276, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18644992, 'num_objects': 2276, 'num_object_clones': 0, 'num_object_copies': 9104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2276, 'num_write_kb': 18208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:02.355 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "27'2113", 'reported_seq': 2164, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.251947+0000', 'last_change': '2024-09-15T21:03:34.737424+0000', 'last_active': '2024-09-15T21:03:52.251947+0000', 'last_peered': '2024-09-15T21:03:52.251947+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:14.812968+0000', 'last_became_peered': '2024-09-15T21:03:14.812968+0000', 'last_unstale': '2024-09-15T21:03:52.251947+0000', 'last_undegraded': '2024-09-15T21:03:14.805983+0000', 'last_fullsized': '2024-09-15T21:03:14.805611+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2113, 'log_dups_size': 0, 'ondisk_log_size': 2113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17274124, 'num_objects': 2109, 'num_object_clones': 0, 'num_object_copies': 8436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2109, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2111, 'num_write_kb': 16884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2109}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2290", 'reported_seq': 2363, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:00.911607+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:00.911607+0000', 'last_peered': '2024-09-15T21:04:00.911607+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:00.911607+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2290, 'log_dups_size': 0, 'ondisk_log_size': 2290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18733702, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2289, 'num_write_kb': 18302, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 29}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "27'2064", 'reported_seq': 2115, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.231179+0000', 'last_change': '2024-09-15T21:03:35.788098+0000', 'last_active': '2024-09-15T21:03:52.231179+0000', 'last_peered': '2024-09-15T21:03:52.231179+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:14.813259+0000', 'last_became_peered': '2024-09-15T21:03:14.813259+0000', 'last_unstale': '2024-09-15T21:03:52.231179+0000', 'last_undegraded': '2024-09-15T21:03:14.803905+0000', 'last_fullsized': '2024-09-15T21:03:14.803589+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2064, 'log_dups_size': 0, 'ondisk_log_size': 2064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16872716, 'num_objects': 2060, 'num_object_clones': 0, 'num_object_copies': 8240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2060, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2060, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2062, 'num_write_kb': 16485, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2060}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "28'2316", 'reported_seq': 2356, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:56.995686+0000', 'last_change': '2024-09-15T21:03:57.047159+0000', 'last_active': '2024-09-15T21:03:57.047159+0000', 'last_peered': '2024-09-15T21:03:56.995686+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:14.818131+0000', 'last_became_peered': '2024-09-15T21:03:14.818131+0000', 'last_unstale': '2024-09-15T21:03:56.995686+0000', 'last_undegraded': '2024-09-15T21:03:14.810890+0000', 'last_fullsized': '2024-09-15T21:03:14.810601+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2316, 'log_dups_size': 0, 'ondisk_log_size': 2316, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18945292, 'num_objects': 2313, 'num_object_clones': 0, 'num_object_copies': 9252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2314, 'num_write_kb': 18509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 1, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "28'2316", 'reported_seq': 2346, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:56.981287+0000', 'last_change': '2024-09-15T21:03:57.047525+0000', 'last_active': '2024-09-15T21:03:57.047525+0000', 'last_peered': '2024-09-15T21:03:56.981287+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:14.814704+0000', 'last_became_peered': '2024-09-15T21:03:14.814704+0000', 'last_unstale': '2024-09-15T21:03:56.981287+0000', 'last_undegraded': '2024-09-15T21:03:23.644925+0000', 'last_fullsized': '2024-09-15T21:03:14.807972+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2316, 'log_dups_size': 0, 'ondisk_log_size': 2316, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18972672, 'num_objects': 2316, 'num_object_clones': 0, 'num_object_copies': 9264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2316, 'num_write_kb': 18528, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "28'2374", 'reported_seq': 2428, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:57.017515+0000', 'last_change': '2024-09-15T21:03:57.043225+0000', 'last_active': '2024-09-15T21:03:57.043225+0000', 'last_peered': '2024-09-15T21:03:57.017515+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:14.812828+0000', 'last_became_peered': '2024-09-15T21:03:14.812828+0000', 'last_unstale': '2024-09-15T21:03:57.017515+0000', 'last_undegraded': '2024-09-15T21:03:14.804528+0000', 'last_fullsized': '2024-09-15T21:03:14.804218+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2374, 'log_dups_size': 0, 'ondisk_log_size': 2374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19395852, 'num_objects': 2368, 'num_object_clones': 0, 'num_object_copies': 9472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2372, 'num_write_kb': 18956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "27'2173", 'reported_seq': 2223, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.235230+0000', 'last_change': '2024-09-15T21:03:37.370975+0000', 'last_active': '2024-09-15T21:03:52.235230+0000', 'last_peered': '2024-09-15T21:03:52.235230+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:14.814661+0000', 'last_became_peered': '2024-09-15T21:03:14.814661+0000', 'last_unstale': '2024-09-15T21:03:52.235230+0000', 'last_undegraded': '2024-09-15T21:03:14.804502+0000', 'last_fullsized': '2024-09-15T21:03:14.804201+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2173, 'log_dups_size': 0, 'ondisk_log_size': 2173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17757452, 'num_objects': 2168, 'num_object_clones': 0, 'num_object_copies': 8672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2168, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2171, 'num_write_kb': 17356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2168}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "28'2344", 'reported_seq': 2388, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:57.016784+0000', 'last_change': '2024-09-15T21:03:57.046439+0000', 'last_active': '2024-09-15T21:03:57.046439+0000', 'last_peered': '2024-09-15T21:03:57.016784+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:14.815069+0000', 'last_became_peered': '2024-09-15T21:03:14.815069+0000', 'last_unstale': '2024-09-15T21:03:57.016784+0000', 'last_undegraded': '2024-09-15T21:03:14.810180+0000', 'last_fullsized': '2024-09-15T21:03:14.809089+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2344, 'log_dups_size': 0, 'ondisk_log_size': 2344, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19130904, 'num_objects': 2336, 'num_object_clones': 0, 'num_object_copies': 9344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2341, 'num_write_kb': 18705, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2364", 'reported_seq': 2439, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:00.777762+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:00.777762+0000', 'last_peered': '2024-09-15T21:04:00.777762+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:00.777762+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2364, 'log_dups_size': 0, 'ondisk_log_size': 2364, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19346700, 'num_objects': 2362, 'num_object_clones': 0, 'num_object_copies': 9448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2362, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2362, 'num_write_kb': 18894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 27}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "28'2268", 'reported_seq': 2312, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-15T21:03:56.999012+0000', 'last_change': '2024-09-15T21:03:57.044257+0000', 'last_active': '2024-09-15T21:03:57.044257+0000', 'last_peered': '2024-09-15T21:03:56.999012+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:14.813569+0000', 'last_became_peered': '2024-09-15T21:03:14.813569+0000', 'last_unstale': '2024-09-15T21:03:56.999012+0000', 'last_undegraded': '2024-09-15T21:03:14.809026+0000', 'last_fullsized': '2024-09-15T21:03:14.808233+0000', 'mapping_epoch': 29, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2268, 'log_dups_size': 0, 'ondisk_log_size': 2268, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18534290, 'num_objects': 2263, 'num_object_clones': 0, 'num_object_copies': 9052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2263, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2265, 'num_write_kb': 18108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "27'1994", 'reported_seq': 2018, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.313143+0000', 'last_change': '2024-09-15T21:03:39.459538+0000', 'last_active': '2024-09-15T21:03:52.313143+0000', 'last_peered': '2024-09-15T21:03:52.313143+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:14.813609+0000', 'last_became_peered': '2024-09-15T21:03:14.813609+0000', 'last_unstale': '2024-09-15T21:03:52.313143+0000', 'last_undegraded': '2024-09-15T21:03:14.808477+0000', 'last_fullsized': '2024-09-15T21:03:14.808198+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 1994, 'log_dups_size': 0, 'ondisk_log_size': 1994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16325254, 'num_objects': 1993, 'num_object_clones': 0, 'num_object_copies': 7972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1993, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 1993, 'num_write_kb': 15943, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1993}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.357 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2318", 'reported_seq': 2392, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:00.896499+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:00.896499+0000', 'last_peered': '2024-09-15T21:04:00.896499+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:00.896499+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2318, 'log_dups_size': 0, 'ondisk_log_size': 2318, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18979462, 'num_objects': 2317, 'num_object_clones': 0, 'num_object_copies': 9268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2317, 'num_write_kb': 18535, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 31}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "27'2028", 'reported_seq': 2067, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.315969+0000', 'last_change': '2024-09-15T21:03:40.593572+0000', 'last_active': '2024-09-15T21:03:52.315969+0000', 'last_peered': '2024-09-15T21:03:52.315969+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:14.816712+0000', 'last_became_peered': '2024-09-15T21:03:14.816712+0000', 'last_unstale': '2024-09-15T21:03:52.315969+0000', 'last_undegraded': '2024-09-15T21:03:14.810156+0000', 'last_fullsized': '2024-09-15T21:03:14.809825+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2028, 'log_dups_size': 0, 'ondisk_log_size': 2028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16577804, 'num_objects': 2024, 'num_object_clones': 0, 'num_object_copies': 8096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2024, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2026, 'num_write_kb': 16204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2024}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2312", 'reported_seq': 2389, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:00.801044+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:00.801044+0000', 'last_peered': '2024-09-15T21:04:00.801044+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:00.801044+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2312, 'log_dups_size': 0, 'ondisk_log_size': 2312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18912524, 'num_objects': 2309, 'num_object_clones': 0, 'num_object_copies': 9236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2310, 'num_write_kb': 18477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 28}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:02.358 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "27'2067", 'reported_seq': 2118, 'reported_epoch': 27, 'state': 'active+undersized+degraded', 'last_fresh': '2024-09-15T21:03:52.209930+0000', 'last_change': '2024-09-15T21:03:41.617853+0000', 'last_active': '2024-09-15T21:03:52.209930+0000', 'last_peered': '2024-09-15T21:03:52.209930+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:14.813091+0000', 'last_became_peered': '2024-09-15T21:03:14.813091+0000', 'last_unstale': '2024-09-15T21:03:52.209930+0000', 'last_undegraded': '2024-09-15T21:03:14.804947+0000', 'last_fullsized': '2024-09-15T21:03:14.804286+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2067, 'log_dups_size': 0, 'ondisk_log_size': 2067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16889100, 'num_objects': 2062, 'num_object_clones': 0, 'num_object_copies': 8248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2062, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2065, 'num_write_kb': 16508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2062}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:02.358 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:02.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 39 20 38180 38160 7.64307 2.97656 0.050449 0.0204263 2024-09-15T21:04:03.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:03.141 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21582/142252 objects degraded (15.172%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":108,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":6},{"state_name":"peering","count":6},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4}],"num_pgs":16,"num_pools":1,"num_objects":35563,"data_bytes":291292840,"bytes_used":937730048,"bytes_avail":385609326592,"bytes_total":386547056640,"inactive_pgs_ratio":0.375,"degraded_objects":21582,"degraded_total":142252,"degraded_ratio":0.15171667182183729,"write_bytes_sec":2590808,"read_op_per_sec":0,"write_op_per_sec":316},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-15T21:03:27.908729+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:03.168 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:04:03.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:03.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:04:03.630461+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0207423 2024-09-15T21:04:03.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:04:03.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 40 20 38553 38533 7.52485 2.91406 0.0498951 0.0207423 2024-09-15T21:04:03.783 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:03.784 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:03.810 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-09-15T21:04:01.917389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":291292840,"num_objects":35563,"num_object_clones":0,"num_object_copies":142252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35590,"num_write_kb":284602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35617,"ondisk_log_size":35617,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":915752,"kb_used_data":446684,"kb_used_omap":14,"kb_used_meta":468977,"kb_avail":376571608,"statfs":{"total":386547056640,"available":385609326592,"internally_reserved":0,"allocated":457404416,"data_stored":456799082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":480233010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,8,12,13],"upper_bound":64},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15302656,"num_objects":1868,"num_object_clones":0,"num_object_copies":7472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-12113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1868,"num_write_kb":14944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.906517"},"pg_stats":[{"pgid":"1.f","version":"28'2276","reported_seq":2307,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.036836+0000","last_change":"2024-09-15T21:03:57.049281+0000","last_active":"2024-09-15T21:03:57.049281+0000","last_peered":"2024-09-15T21:03:57.036836+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814722+0000","last_became_peered":"2024-09-15T21:03:14.814722+0000","last_unstale":"2024-09-15T21:03:57.036836+0000","last_undegraded":"2024-09-15T21:03:23.636203+0000","last_fullsized":"2024-09-15T21:03:14.810067+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18644992,"num_objects":2276,"num_object_clones":0,"num_object_copies":9104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2276,"num_write_kb":18208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"27'2113","reported_seq":2164,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.251947+0000","last_change":"2024-09-15T21:03:34.737424+0000","last_active":"2024-09-15T21:03:52.251947+0000","last_peered":"2024-09-15T21:03:52.251947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:14.812968+0000","last_became_peered":"2024-09-15T21:03:14.812968+0000","last_unstale":"2024-09-15T21:03:52.251947+0000","last_undegraded":"2024-09-15T21:03:14.805983+0000","last_fullsized":"2024-09-15T21:03:14.805611+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:35.452953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17274124,"num_objects":2109,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":16884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2109}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2290","reported_seq":2363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.911607+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:00.911607+0000","last_peered":"2024-09-15T21:04:00.911607+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:00.911607+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2290,"log_dups_size":0,"ondisk_log_size":2290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18733702,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'2064","reported_seq":2115,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.231179+0000","last_change":"2024-09-15T21:03:35.788098+0000","last_active":"2024-09-15T21:03:52.231179+0000","last_peered":"2024-09-15T21:03:52.231179+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:14.813259+0000","last_became_peered":"2024-09-15T21:03:14.813259+0000","last_unstale":"2024-09-15T21:03:52.231179+0000","last_undegraded":"2024-09-15T21:03:14.803905+0000","last_fullsized":"2024-09-15T21:03:14.803589+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:36.444615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16872716,"num_objects":2060,"num_object_clones":0,"num_object_copies":8240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2060,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2062,"num_write_kb":16485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2060}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"28'2316","reported_seq":2356,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.995686+0000","last_change":"2024-09-15T21:03:57.047159+0000","last_active":"2024-09-15T21:03:57.047159+0000","last_peered":"2024-09-15T21:03:56.995686+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:14.818131+0000","last_became_peered":"2024-09-15T21:03:14.818131+0000","last_unstale":"2024-09-15T21:03:56.995686+0000","last_undegraded":"2024-09-15T21:03:14.810890+0000","last_fullsized":"2024-09-15T21:03:14.810601+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":18509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"28'2316","reported_seq":2346,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.981287+0000","last_change":"2024-09-15T21:03:57.047525+0000","last_active":"2024-09-15T21:03:57.047525+0000","last_peered":"2024-09-15T21:03:56.981287+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:14.814704+0000","last_became_peered":"2024-09-15T21:03:14.814704+0000","last_unstale":"2024-09-15T21:03:56.981287+0000","last_undegraded":"2024-09-15T21:03:23.644925+0000","last_fullsized":"2024-09-15T21:03:14.807972+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2316,"log_dups_size":0,"ondisk_log_size":2316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18972672,"num_objects":2316,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2316,"num_write_kb":18528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"28'2374","reported_seq":2428,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.017515+0000","last_change":"2024-09-15T21:03:57.043225+0000","last_active":"2024-09-15T21:03:57.043225+0000","last_peered":"2024-09-15T21:03:57.017515+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:14.812828+0000","last_became_peered":"2024-09-15T21:03:14.812828+0000","last_unstale":"2024-09-15T21:03:57.017515+0000","last_undegraded":"2024-09-15T21:03:14.804528+0000","last_fullsized":"2024-09-15T21:03:14.804218+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19395852,"num_objects":2368,"num_object_clones":0,"num_object_copies":9472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":18956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"27'2173","reported_seq":2223,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.235230+0000","last_change":"2024-09-15T21:03:37.370975+0000","last_active":"2024-09-15T21:03:52.235230+0000","last_peered":"2024-09-15T21:03:52.235230+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:14.814661+0000","last_became_peered":"2024-09-15T21:03:14.814661+0000","last_unstale":"2024-09-15T21:03:52.235230+0000","last_undegraded":"2024-09-15T21:03:14.804502+0000","last_fullsized":"2024-09-15T21:03:14.804201+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:38.503399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":17356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2168}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"28'2344","reported_seq":2388,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:57.016784+0000","last_change":"2024-09-15T21:03:57.046439+0000","last_active":"2024-09-15T21:03:57.046439+0000","last_peered":"2024-09-15T21:03:57.016784+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:14.815069+0000","last_became_peered":"2024-09-15T21:03:14.815069+0000","last_unstale":"2024-09-15T21:03:57.016784+0000","last_undegraded":"2024-09-15T21:03:14.810180+0000","last_fullsized":"2024-09-15T21:03:14.809089+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2344,"log_dups_size":0,"ondisk_log_size":2344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19130904,"num_objects":2336,"num_object_clones":0,"num_object_copies":9344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2341,"num_write_kb":18705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2364","reported_seq":2439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.777762+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:00.777762+0000","last_peered":"2024-09-15T21:04:00.777762+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:00.777762+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19346700,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"28'2268","reported_seq":2312,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-15T21:03:56.999012+0000","last_change":"2024-09-15T21:03:57.044257+0000","last_active":"2024-09-15T21:03:57.044257+0000","last_peered":"2024-09-15T21:03:56.999012+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:14.813569+0000","last_became_peered":"2024-09-15T21:03:14.813569+0000","last_unstale":"2024-09-15T21:03:56.999012+0000","last_undegraded":"2024-09-15T21:03:14.809026+0000","last_fullsized":"2024-09-15T21:03:14.808233+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2268,"log_dups_size":0,"ondisk_log_size":2268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18534290,"num_objects":2263,"num_object_clones":0,"num_object_copies":9052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2265,"num_write_kb":18108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"27'1994","reported_seq":2018,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.313143+0000","last_change":"2024-09-15T21:03:39.459538+0000","last_active":"2024-09-15T21:03:52.313143+0000","last_peered":"2024-09-15T21:03:52.313143+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:14.813609+0000","last_became_peered":"2024-09-15T21:03:14.813609+0000","last_unstale":"2024-09-15T21:03:52.313143+0000","last_undegraded":"2024-09-15T21:03:14.808477+0000","last_fullsized":"2024-09-15T21:03:14.808198+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":1994,"log_dups_size":0,"ondisk_log_size":1994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:40.939169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16325254,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1993,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":15943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1993}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2318","reported_seq":2392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.896499+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:00.896499+0000","last_peered":"2024-09-15T21:04:00.896499+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:00.896499+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2318,"log_dups_size":0,"ondisk_log_size":2318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18979462,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":18535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"27'2028","reported_seq":2067,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.315969+0000","last_change":"2024-09-15T21:03:40.593572+0000","last_active":"2024-09-15T21:03:52.315969+0000","last_peered":"2024-09-15T21:03:52.315969+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:14.816712+0000","last_became_peered":"2024-09-15T21:03:14.816712+0000","last_unstale":"2024-09-15T21:03:52.315969+0000","last_undegraded":"2024-09-15T21:03:14.810156+0000","last_fullsized":"2024-09-15T21:03:14.809825+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2028,"log_dups_size":0,"ondisk_log_size":2028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:41.970502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16577804,"num_objects":2024,"num_object_clones":0,"num_object_copies":8096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":16204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2312","reported_seq":2389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.801044+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:00.801044+0000","last_peered":"2024-09-15T21:04:00.801044+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:00.801044+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912524,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":18477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"27'2067","reported_seq":2118,"reported_epoch":27,"state":"active+undersized+degraded","last_fresh":"2024-09-15T21:03:52.209930+0000","last_change":"2024-09-15T21:03:41.617853+0000","last_active":"2024-09-15T21:03:52.209930+0000","last_peered":"2024-09-15T21:03:52.209930+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:14.813091+0000","last_became_peered":"2024-09-15T21:03:14.813091+0000","last_unstale":"2024-09-15T21:03:52.209930+0000","last_undegraded":"2024-09-15T21:03:14.804947+0000","last_fullsized":"2024-09-15T21:03:14.804286+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-15T21:04:42.481339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16889100,"num_objects":2062,"num_object_clones":0,"num_object_copies":8248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":16508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2062}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":291292840,"num_objects":35563,"num_object_clones":0,"num_object_copies":142252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35590,"num_write_kb":284602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459636736,"data_stored":459636736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35617,"ondisk_log_size":35617,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288060,"kb_used_data":148072,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94083780,"statfs":{"total":96636764160,"available":96341790720,"internally_reserved":0,"allocated":151625728,"data_stored":151480011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,3,5,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293732,"kb_used_data":150100,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94078108,"statfs":{"total":96636764160,"available":96335982592,"internally_reserved":0,"allocated":153702400,"data_stored":153545450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51056,"kb_used_data":1372,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94320784,"statfs":{"total":96636764160,"available":96584482816,"internally_reserved":0,"allocated":1404928,"data_stored":1247978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":282904,"kb_used_data":147140,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94088936,"statfs":{"total":96636764160,"available":96347070464,"internally_reserved":0,"allocated":150671360,"data_stored":150525643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,4,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2236416,"data_stored":2236416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153804800,"data_stored":153804800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151797760,"data_stored":151797760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:04.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 41 20 38975 38955 7.42172 3.29688 0.0560292 0.0210376 2024-09-15T21:04:05.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 42 20 39400 39380 7.32407 3.32031 0.0514014 0.0213181 2024-09-15T21:04:06.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 43 20 39817 39797 7.22951 3.25781 0.0930784 0.0215878 2024-09-15T21:04:06.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:07.430 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:07.431 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:07.460 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-15T21:04:05.918320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":310953640,"num_objects":37963,"num_object_clones":0,"num_object_copies":151852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37990,"num_write_kb":303802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":204800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38017,"ondisk_log_size":38017,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":924048,"kb_used_data":454984,"kb_used_omap":14,"kb_used_meta":468977,"kb_avail":376563312,"statfs":{"total":386547056640,"available":385600831488,"internally_reserved":0,"allocated":465903616,"data_stored":465275816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":480233010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,8,26,11,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22487040,"num_objects":2745,"num_object_clones":0,"num_object_copies":10980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2745,"num_write_kb":21960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":204800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001722"},"pg_stats":[{"pgid":"1.f","version":"31'2343","reported_seq":2382,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.432269+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:02.432269+0000","last_peered":"2024-09-15T21:04:02.432269+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:02.432269+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2343,"log_dups_size":0,"ondisk_log_size":2343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19193856,"num_objects":2343,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":18744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":67}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2448","reported_seq":2510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.240270+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:02.240270+0000","last_peered":"2024-09-15T21:04:02.240270+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:02.240270+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2448,"log_dups_size":0,"ondisk_log_size":2448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20018444,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2446,"num_write_kb":19564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2290","reported_seq":2363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.911607+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:00.911607+0000","last_peered":"2024-09-15T21:04:00.911607+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:00.911607+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2290,"log_dups_size":0,"ondisk_log_size":2290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18733702,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2289,"num_write_kb":18302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":29}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2404","reported_seq":2467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.254709+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:02.254709+0000","last_peered":"2024-09-15T21:04:02.254709+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:02.254709+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2404,"log_dups_size":0,"ondisk_log_size":2404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19657996,"num_objects":2400,"num_object_clones":0,"num_object_copies":9600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2402,"num_write_kb":19205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":66}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2382","reported_seq":2431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.403290+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:02.403290+0000","last_peered":"2024-09-15T21:04:02.403290+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:02.403290+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2382,"log_dups_size":0,"ondisk_log_size":2382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19485964,"num_objects":2379,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2380,"num_write_kb":19037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":68}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2372","reported_seq":2411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.378395+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:02.378395+0000","last_peered":"2024-09-15T21:04:02.378395+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:02.378395+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19431424,"num_objects":2372,"num_object_clones":0,"num_object_copies":9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":18976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":56}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2431","reported_seq":2495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.179972+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:02.179972+0000","last_peered":"2024-09-15T21:04:02.179972+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:02.179972+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2431,"log_dups_size":0,"ondisk_log_size":2431,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19862796,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":19412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":59}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2550","reported_seq":2613,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.256373+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:02.256373+0000","last_peered":"2024-09-15T21:04:02.256373+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:02.256373+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2550,"log_dups_size":0,"ondisk_log_size":2550,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845836,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":20372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":71}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2415","reported_seq":2469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.419567+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:02.419567+0000","last_peered":"2024-09-15T21:04:02.419567+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:02.419567+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":0,"ondisk_log_size":2415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19712536,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2412,"num_write_kb":19273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":75}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2364","reported_seq":2439,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.777762+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:00.777762+0000","last_peered":"2024-09-15T21:04:00.777762+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:00.777762+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2364,"log_dups_size":0,"ondisk_log_size":2364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19346700,"num_objects":2362,"num_object_clones":0,"num_object_copies":9448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2362,"num_write_kb":18894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":27}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2339","reported_seq":2444,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.431813+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:02.431813+0000","last_peered":"2024-09-15T21:04:02.431813+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:02.431813+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2339,"log_dups_size":0,"ondisk_log_size":2339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19115922,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2336,"num_write_kb":18676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":204800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":99}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2315","reported_seq":2348,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.391187+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:02.391187+0000","last_peered":"2024-09-15T21:04:02.391187+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:02.391187+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2315,"log_dups_size":0,"ondisk_log_size":2315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18954886,"num_objects":2314,"num_object_clones":0,"num_object_copies":9256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2314,"num_write_kb":18511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2318","reported_seq":2392,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.896499+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:00.896499+0000","last_peered":"2024-09-15T21:04:00.896499+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:00.896499+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2318,"log_dups_size":0,"ondisk_log_size":2318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18979462,"num_objects":2317,"num_object_clones":0,"num_object_copies":9268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2317,"num_write_kb":18535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2342","reported_seq":2393,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.388898+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:02.388898+0000","last_peered":"2024-09-15T21:04:02.388898+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:02.388898+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2342,"log_dups_size":0,"ondisk_log_size":2342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19150092,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2340,"num_write_kb":18716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":59}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2312","reported_seq":2389,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:00.801044+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:00.801044+0000","last_peered":"2024-09-15T21:04:00.801044+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:00.801044+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2312,"log_dups_size":0,"ondisk_log_size":2312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18912524,"num_objects":2309,"num_object_clones":0,"num_object_copies":9236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":18477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":28}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2392","reported_seq":2454,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:02.243194+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:02.243194+0000","last_peered":"2024-09-15T21:04:02.243194+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:02.243194+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2392,"log_dups_size":0,"ondisk_log_size":2392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19551500,"num_objects":2387,"num_object_clones":0,"num_object_copies":9548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2390,"num_write_kb":19108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":72}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":310953640,"num_objects":37963,"num_object_clones":0,"num_object_copies":151852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":37058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":37990,"num_write_kb":303802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":204800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":467988480,"data_stored":467988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38017,"ondisk_log_size":38017,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291896,"kb_used_data":151912,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94079944,"statfs":{"total":96636764160,"available":96337862656,"internally_reserved":0,"allocated":155557888,"data_stored":155400938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,14,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293732,"kb_used_data":150100,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":94078108,"statfs":{"total":96636764160,"available":96335982592,"internally_reserved":0,"allocated":153702400,"data_stored":153545450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051586,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51056,"kb_used_data":1372,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94320784,"statfs":{"total":96636764160,"available":96584482816,"internally_reserved":0,"allocated":1404928,"data_stored":1247978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,3,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287364,"kb_used_data":151600,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94084476,"statfs":{"total":96636764160,"available":96342503424,"internally_reserved":0,"allocated":155238400,"data_stored":155081450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,8,3,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":155856896,"data_stored":155856896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2236416,"data_stored":2236416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153804800,"data_stored":153804800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":156090368,"data_stored":156090368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:07.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:07.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 44 20 40199 40179 7.13303 2.98438 0.0506525 0.0218871 2024-09-15T21:04:08.033 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:08.033 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:08.067 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-15T21:04:07.918754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327558824,"num_objects":39990,"num_object_clones":0,"num_object_copies":159960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40017,"num_write_kb":320018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40044,"ondisk_log_size":40044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":980840,"kb_used_data":486824,"kb_used_omap":14,"kb_used_meta":493937,"kb_avail":376506520,"statfs":{"total":386547056640,"available":385542676480,"internally_reserved":0,"allocated":498507776,"data_stored":497879976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":505792050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,6,17,18,7,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39092224,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.893403"},"pg_stats":[{"pgid":"1.f","version":"31'2452","reported_seq":2490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.460441+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:07.460441+0000","last_peered":"2024-09-15T21:04:07.460441+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:07.460441+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2575","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.251738+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:07.251738+0000","last_peered":"2024-09-15T21:04:07.251738+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:07.251738+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058828,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2411","reported_seq":2483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.994580+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:05.994580+0000","last_peered":"2024-09-15T21:04:05.994580+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:05.994580+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19724934,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":19270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2526","reported_seq":2589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.207045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:07.207045+0000","last_peered":"2024-09-15T21:04:07.207045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:07.207045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657420,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":20181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2505","reported_seq":2555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.412894+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:07.412894+0000","last_peered":"2024-09-15T21:04:07.412894+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:07.412894+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493580,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2499","reported_seq":2538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.469983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:07.469983+0000","last_peered":"2024-09-15T21:04:07.469983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:07.469983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2552","reported_seq":2616,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.268281+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:07.268281+0000","last_peered":"2024-09-15T21:04:07.268281+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:07.268281+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2552,"log_dups_size":0,"ondisk_log_size":2552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854028,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2658","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.155317+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:07.155317+0000","last_peered":"2024-09-15T21:04:07.155317+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:07.155317+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2658,"log_dups_size":0,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730572,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2534","reported_seq":2588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382738+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:07.382738+0000","last_peered":"2024-09-15T21:04:07.382738+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:07.382738+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2534,"log_dups_size":0,"ondisk_log_size":2534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20687384,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2511","reported_seq":2586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.954459+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:05.954459+0000","last_peered":"2024-09-15T21:04:05.954459+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:05.954459+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2511,"log_dups_size":0,"ondisk_log_size":2511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550924,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":20070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2471","reported_seq":2700,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.442186+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:07.442186+0000","last_peered":"2024-09-15T21:04:07.442186+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:07.442186+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20197266,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":19732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2445","reported_seq":2478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.468081+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:07.468081+0000","last_peered":"2024-09-15T21:04:07.468081+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:07.468081+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20019846,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":19551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2455","reported_seq":2529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.785931+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:05.785931+0000","last_peered":"2024-09-15T21:04:05.785931+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:05.785931+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20101766,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2473","reported_seq":2524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382442+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:07.382442+0000","last_peered":"2024-09-15T21:04:07.382442+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:07.382442+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20223244,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2441","reported_seq":2516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.985286+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:05.985286+0000","last_peered":"2024-09-15T21:04:05.985286+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:05.985286+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19969292,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2536","reported_seq":2599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.292801+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:07.292801+0000","last_peered":"2024-09-15T21:04:07.292801+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:07.292801+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731148,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":327558824,"num_objects":39990,"num_object_clones":0,"num_object_copies":159960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40017,"num_write_kb":320018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501366784,"data_stored":501366784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40044,"ondisk_log_size":40044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308300,"kb_used_data":159996,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94063540,"statfs":{"total":96636764160,"available":96321064960,"internally_reserved":0,"allocated":163835904,"data_stored":163678954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,4,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305888,"kb_used_data":158096,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94065952,"statfs":{"total":96636764160,"available":96323534848,"internally_reserved":0,"allocated":161890304,"data_stored":161733354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,0,2,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63056,"kb_used_data":9216,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94308784,"statfs":{"total":96636764160,"available":96572194816,"internally_reserved":0,"allocated":9437184,"data_stored":9280234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55111375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303596,"kb_used_data":159516,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068244,"statfs":{"total":96636764160,"available":96325881856,"internally_reserved":0,"allocated":163344384,"data_stored":163187434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164200448,"data_stored":164200448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10694656,"data_stored":10694656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162017280,"data_stored":162017280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164454400,"data_stored":164454400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2452", 'reported_seq': 2490, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.460441+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:07.460441+0000', 'last_peered': '2024-09-15T21:04:07.460441+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:07.460441+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2452, 'log_dups_size': 0, 'ondisk_log_size': 2452, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20086784, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2452, 'num_write_kb': 19616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 176}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2575", 'reported_seq': 2637, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.251738+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:07.251738+0000', 'last_peered': '2024-09-15T21:04:07.251738+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:07.251738+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2575, 'log_dups_size': 0, 'ondisk_log_size': 2575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21058828, 'num_objects': 2571, 'num_object_clones': 0, 'num_object_copies': 10284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2573, 'num_write_kb': 20580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:08.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2411", 'reported_seq': 2483, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:05.994580+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:05.994580+0000', 'last_peered': '2024-09-15T21:04:05.994580+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:05.994580+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2411, 'log_dups_size': 0, 'ondisk_log_size': 2411, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19724934, 'num_objects': 2408, 'num_object_clones': 0, 'num_object_copies': 9632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2410, 'num_write_kb': 19270, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 150}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2526", 'reported_seq': 2589, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.207045+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:07.207045+0000', 'last_peered': '2024-09-15T21:04:07.207045+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:07.207045+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2526, 'log_dups_size': 0, 'ondisk_log_size': 2526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657420, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2524, 'num_write_kb': 20181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2505", 'reported_seq': 2555, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.412894+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:07.412894+0000', 'last_peered': '2024-09-15T21:04:07.412894+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:07.412894+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2505, 'log_dups_size': 0, 'ondisk_log_size': 2505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20493580, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 10008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2503, 'num_write_kb': 20021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2499", 'reported_seq': 2538, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.469983+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:07.469983+0000', 'last_peered': '2024-09-15T21:04:07.469983+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:07.469983+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2499, 'log_dups_size': 0, 'ondisk_log_size': 2499, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20471808, 'num_objects': 2499, 'num_object_clones': 0, 'num_object_copies': 9996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2499, 'num_write_kb': 19992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2552", 'reported_seq': 2616, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.268281+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:07.268281+0000', 'last_peered': '2024-09-15T21:04:07.268281+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:07.268281+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2552, 'log_dups_size': 0, 'ondisk_log_size': 2552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20854028, 'num_objects': 2546, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2550, 'num_write_kb': 20380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 180}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2658", 'reported_seq': 2721, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.155317+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:07.155317+0000', 'last_peered': '2024-09-15T21:04:07.155317+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:07.155317+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2658, 'log_dups_size': 0, 'ondisk_log_size': 2658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21730572, 'num_objects': 2653, 'num_object_clones': 0, 'num_object_copies': 10612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2656, 'num_write_kb': 21236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2534", 'reported_seq': 2588, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.382738+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:07.382738+0000', 'last_peered': '2024-09-15T21:04:07.382738+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:07.382738+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2534, 'log_dups_size': 0, 'ondisk_log_size': 2534, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20687384, 'num_objects': 2526, 'num_object_clones': 0, 'num_object_copies': 10104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2531, 'num_write_kb': 20225, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 194}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2511", 'reported_seq': 2586, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:05.954459+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:05.954459+0000', 'last_peered': '2024-09-15T21:04:05.954459+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:05.954459+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2511, 'log_dups_size': 0, 'ondisk_log_size': 2511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20550924, 'num_objects': 2509, 'num_object_clones': 0, 'num_object_copies': 10036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2509, 'num_write_kb': 20070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 174}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2471", 'reported_seq': 2700, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.442186+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:07.442186+0000', 'last_peered': '2024-09-15T21:04:07.442186+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:07.442186+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2471, 'log_dups_size': 0, 'ondisk_log_size': 2471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20197266, 'num_objects': 2466, 'num_object_clones': 0, 'num_object_copies': 9864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2172, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2468, 'num_write_kb': 19732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 720896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2445", 'reported_seq': 2478, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.468081+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:07.468081+0000', 'last_peered': '2024-09-15T21:04:07.468081+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:07.468081+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2445, 'log_dups_size': 0, 'ondisk_log_size': 2445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20019846, 'num_objects': 2444, 'num_object_clones': 0, 'num_object_copies': 9776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2444, 'num_write_kb': 19551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2455", 'reported_seq': 2529, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:05.785931+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:05.785931+0000', 'last_peered': '2024-09-15T21:04:05.785931+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:05.785931+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2455, 'log_dups_size': 0, 'ondisk_log_size': 2455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20101766, 'num_objects': 2454, 'num_object_clones': 0, 'num_object_copies': 9816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2454, 'num_write_kb': 19631, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2473", 'reported_seq': 2524, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.382442+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:07.382442+0000', 'last_peered': '2024-09-15T21:04:07.382442+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:07.382442+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2473, 'log_dups_size': 0, 'ondisk_log_size': 2473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20223244, 'num_objects': 2469, 'num_object_clones': 0, 'num_object_copies': 9876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2471, 'num_write_kb': 19764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:08.070 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:08.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2441", 'reported_seq': 2516, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:05.985286+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:05.985286+0000', 'last_peered': '2024-09-15T21:04:05.985286+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:05.985286+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2441, 'log_dups_size': 0, 'ondisk_log_size': 2441, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19969292, 'num_objects': 2438, 'num_object_clones': 0, 'num_object_copies': 9752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2438, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2439, 'num_write_kb': 19509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 157}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:08.071 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:08.071 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2536", 'reported_seq': 2599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.292801+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:07.292801+0000', 'last_peered': '2024-09-15T21:04:07.292801+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:07.292801+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2536, 'log_dups_size': 0, 'ondisk_log_size': 2536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20731148, 'num_objects': 2531, 'num_object_clones': 0, 'num_object_copies': 10124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2534, 'num_write_kb': 20260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:08.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:08.632 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 45 20 40565 40545 7.03806 2.85938 0.0736002 0.0221849 2024-09-15T21:04:08.785 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:08.785 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36995/159960 objects degraded (23.128%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":114,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39990,"data_bytes":327558824,"bytes_used":1004380160,"bytes_avail":385542676480,"bytes_total":386547056640,"degraded_objects":36995,"degraded_total":159960,"degraded_ratio":0.23127656914228556,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":72866,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"write_bytes_sec":3951342,"read_op_per_sec":0,"write_op_per_sec":482},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-15T21:04:03.917924+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:08.814 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:08.814 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:09.384 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:09.384 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:09.414 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-15T21:04:07.918754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":327558824,"num_objects":39990,"num_object_clones":0,"num_object_copies":159960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40017,"num_write_kb":320018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40044,"ondisk_log_size":40044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":980840,"kb_used_data":486824,"kb_used_omap":14,"kb_used_meta":493937,"kb_avail":376506520,"statfs":{"total":386547056640,"available":385542676480,"internally_reserved":0,"allocated":498507776,"data_stored":497879976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":505792050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,6,17,18,7,17,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39092224,"num_objects":4772,"num_object_clones":0,"num_object_copies":19088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":38176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.893403"},"pg_stats":[{"pgid":"1.f","version":"31'2452","reported_seq":2490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.460441+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:07.460441+0000","last_peered":"2024-09-15T21:04:07.460441+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:07.460441+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2575","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.251738+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:07.251738+0000","last_peered":"2024-09-15T21:04:07.251738+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:07.251738+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058828,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2411","reported_seq":2483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.994580+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:05.994580+0000","last_peered":"2024-09-15T21:04:05.994580+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:05.994580+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2411,"log_dups_size":0,"ondisk_log_size":2411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19724934,"num_objects":2408,"num_object_clones":0,"num_object_copies":9632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":19270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":150}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2526","reported_seq":2589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.207045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:07.207045+0000","last_peered":"2024-09-15T21:04:07.207045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:07.207045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657420,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":20181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2505","reported_seq":2555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.412894+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:07.412894+0000","last_peered":"2024-09-15T21:04:07.412894+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:07.412894+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493580,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2499","reported_seq":2538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.469983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:07.469983+0000","last_peered":"2024-09-15T21:04:07.469983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:07.469983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2552","reported_seq":2616,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.268281+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:07.268281+0000","last_peered":"2024-09-15T21:04:07.268281+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:07.268281+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2552,"log_dups_size":0,"ondisk_log_size":2552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854028,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2658","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.155317+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:07.155317+0000","last_peered":"2024-09-15T21:04:07.155317+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:07.155317+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2658,"log_dups_size":0,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730572,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2534","reported_seq":2588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382738+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:07.382738+0000","last_peered":"2024-09-15T21:04:07.382738+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:07.382738+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2534,"log_dups_size":0,"ondisk_log_size":2534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20687384,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2511","reported_seq":2586,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.954459+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:05.954459+0000","last_peered":"2024-09-15T21:04:05.954459+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:05.954459+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2511,"log_dups_size":0,"ondisk_log_size":2511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20550924,"num_objects":2509,"num_object_clones":0,"num_object_copies":10036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2509,"num_write_kb":20070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":174}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2471","reported_seq":2700,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.442186+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:07.442186+0000","last_peered":"2024-09-15T21:04:07.442186+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:07.442186+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20197266,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":19732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2445","reported_seq":2478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.468081+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:07.468081+0000","last_peered":"2024-09-15T21:04:07.468081+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:07.468081+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20019846,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":19551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2455","reported_seq":2529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.785931+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:05.785931+0000","last_peered":"2024-09-15T21:04:05.785931+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:05.785931+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2455,"log_dups_size":0,"ondisk_log_size":2455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20101766,"num_objects":2454,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":19631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2473","reported_seq":2524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382442+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:07.382442+0000","last_peered":"2024-09-15T21:04:07.382442+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:07.382442+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20223244,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2441","reported_seq":2516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:05.985286+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:05.985286+0000","last_peered":"2024-09-15T21:04:05.985286+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:05.985286+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2441,"log_dups_size":0,"ondisk_log_size":2441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19969292,"num_objects":2438,"num_object_clones":0,"num_object_copies":9752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":19509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":157}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2536","reported_seq":2599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.292801+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:07.292801+0000","last_peered":"2024-09-15T21:04:07.292801+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:07.292801+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731148,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":327558824,"num_objects":39990,"num_object_clones":0,"num_object_copies":159960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40017,"num_write_kb":320018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501366784,"data_stored":501366784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40044,"ondisk_log_size":40044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308300,"kb_used_data":159996,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94063540,"statfs":{"total":96636764160,"available":96321064960,"internally_reserved":0,"allocated":163835904,"data_stored":163678954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,4,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305888,"kb_used_data":158096,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94065952,"statfs":{"total":96636764160,"available":96323534848,"internally_reserved":0,"allocated":161890304,"data_stored":161733354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,0,2,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63056,"kb_used_data":9216,"kb_used_omap":4,"kb_used_meta":53819,"kb_avail":94308784,"statfs":{"total":96636764160,"available":96572194816,"internally_reserved":0,"allocated":9437184,"data_stored":9280234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":55111375},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,7,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303596,"kb_used_data":159516,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068244,"statfs":{"total":96636764160,"available":96325881856,"internally_reserved":0,"allocated":163344384,"data_stored":163187434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164200448,"data_stored":164200448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10694656,"data_stored":10694656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":162017280,"data_stored":162017280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164454400,"data_stored":164454400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:09.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 46 20 40926 40906 6.94637 2.82031 0.0580162 0.0224711 2024-09-15T21:04:10.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 47 20 41328 41308 6.86539 3.14062 0.0406539 0.0227371 2024-09-15T21:04:11.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 48 20 41732 41712 6.78812 3.15625 0.046743 0.0229897 2024-09-15T21:04:12.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:12.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 49 20 42136 42116 6.714 3.15625 0.0618331 0.0232541 2024-09-15T21:04:12.993 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:12.994 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:13.023 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-15T21:04:11.919718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331343528,"num_objects":40452,"num_object_clones":0,"num_object_copies":161808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40479,"num_write_kb":323714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40506,"ondisk_log_size":40506,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":994008,"kb_used_data":504152,"kb_used_omap":14,"kb_used_meta":489777,"kb_avail":376493352,"statfs":{"total":386547056640,"available":385529192448,"internally_reserved":0,"allocated":516251648,"data_stored":515623848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":501532210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,13,18,6,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42876928,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002715"},"pg_stats":[{"pgid":"1.f","version":"31'2452","reported_seq":2490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.460441+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:07.460441+0000","last_peered":"2024-09-15T21:04:07.460441+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:07.460441+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2575","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.251738+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:07.251738+0000","last_peered":"2024-09-15T21:04:07.251738+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:07.251738+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058828,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2504","reported_seq":2577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.950011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:10.950011+0000","last_peered":"2024-09-15T21:04:10.950011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:10.950011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2504,"log_dups_size":0,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20486790,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2526","reported_seq":2589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.207045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:07.207045+0000","last_peered":"2024-09-15T21:04:07.207045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:07.207045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657420,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":20181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2505","reported_seq":2555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.412894+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:07.412894+0000","last_peered":"2024-09-15T21:04:07.412894+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:07.412894+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493580,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2499","reported_seq":2538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.469983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:07.469983+0000","last_peered":"2024-09-15T21:04:07.469983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:07.469983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2552","reported_seq":2616,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.268281+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:07.268281+0000","last_peered":"2024-09-15T21:04:07.268281+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:07.268281+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2552,"log_dups_size":0,"ondisk_log_size":2552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854028,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2658","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.155317+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:07.155317+0000","last_peered":"2024-09-15T21:04:07.155317+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:07.155317+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2658,"log_dups_size":0,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730572,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2534","reported_seq":2588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382738+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:07.382738+0000","last_peered":"2024-09-15T21:04:07.382738+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:07.382738+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2534,"log_dups_size":0,"ondisk_log_size":2534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20687384,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2631","reported_seq":2705,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.953268+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:10.953268+0000","last_peered":"2024-09-15T21:04:10.953268+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:10.953268+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21533964,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2471","reported_seq":2700,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.442186+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:07.442186+0000","last_peered":"2024-09-15T21:04:07.442186+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:07.442186+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20197266,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":19732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2445","reported_seq":2478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.468081+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:07.468081+0000","last_peered":"2024-09-15T21:04:07.468081+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:07.468081+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20019846,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":19551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2569","reported_seq":2643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.954341+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:10.954341+0000","last_peered":"2024-09-15T21:04:10.954341+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:10.954341+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21035654,"num_objects":2568,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2568,"num_write_kb":20543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2473","reported_seq":2524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382442+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:07.382442+0000","last_peered":"2024-09-15T21:04:07.382442+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:07.382442+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20223244,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2576","reported_seq":2653,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.998516+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:10.998516+0000","last_peered":"2024-09-15T21:04:10.998516+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:10.998516+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21075212,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2536","reported_seq":2599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.292801+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:07.292801+0000","last_peered":"2024-09-15T21:04:07.292801+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:07.292801+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731148,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331343528,"num_objects":40452,"num_object_clones":0,"num_object_copies":161808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40479,"num_write_kb":323714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517484544,"data_stored":517484544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40506,"ondisk_log_size":40506,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308300,"kb_used_data":159996,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94063540,"statfs":{"total":96636764160,"available":96321064960,"internally_reserved":0,"allocated":163835904,"data_stored":163678954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,4,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301280,"kb_used_data":165968,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94070560,"statfs":{"total":96636764160,"available":96328253440,"internally_reserved":0,"allocated":169951232,"data_stored":169794282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80832,"kb_used_data":18672,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94291008,"statfs":{"total":96636764160,"available":96553992192,"internally_reserved":0,"allocated":19120128,"data_stored":18963178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63631055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303596,"kb_used_data":159516,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068244,"statfs":{"total":96636764160,"available":96325881856,"internally_reserved":0,"allocated":163344384,"data_stored":163187434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164200448,"data_stored":164200448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18890752,"data_stored":18890752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169938944,"data_stored":169938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164454400,"data_stored":164454400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:13.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:13.595 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:13.596 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:13.621 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-15T21:04:11.919718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":331343528,"num_objects":40452,"num_object_clones":0,"num_object_copies":161808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40479,"num_write_kb":323714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40506,"ondisk_log_size":40506,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":994008,"kb_used_data":504152,"kb_used_omap":14,"kb_used_meta":489777,"kb_avail":376493352,"statfs":{"total":386547056640,"available":385529192448,"internally_reserved":0,"allocated":516251648,"data_stored":515623848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":501532210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,13,18,6,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42876928,"num_objects":5234,"num_object_clones":0,"num_object_copies":20936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5234,"num_write_kb":41872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002715"},"pg_stats":[{"pgid":"1.f","version":"31'2452","reported_seq":2490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.460441+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:07.460441+0000","last_peered":"2024-09-15T21:04:07.460441+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:07.460441+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":19616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":176}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2575","reported_seq":2637,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.251738+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:07.251738+0000","last_peered":"2024-09-15T21:04:07.251738+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:07.251738+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2575,"log_dups_size":0,"ondisk_log_size":2575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21058828,"num_objects":2571,"num_object_clones":0,"num_object_copies":10284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":20580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":191}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2504","reported_seq":2577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.950011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:10.950011+0000","last_peered":"2024-09-15T21:04:10.950011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:10.950011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2504,"log_dups_size":0,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20486790,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2526","reported_seq":2589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.207045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:07.207045+0000","last_peered":"2024-09-15T21:04:07.207045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:07.207045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2526,"log_dups_size":0,"ondisk_log_size":2526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20657420,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":20181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2505","reported_seq":2555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.412894+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:07.412894+0000","last_peered":"2024-09-15T21:04:07.412894+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:07.412894+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20493580,"num_objects":2502,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2499","reported_seq":2538,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.469983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:07.469983+0000","last_peered":"2024-09-15T21:04:07.469983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:07.469983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2499,"log_dups_size":0,"ondisk_log_size":2499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20471808,"num_objects":2499,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2499,"num_write_kb":19992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2552","reported_seq":2616,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.268281+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:07.268281+0000","last_peered":"2024-09-15T21:04:07.268281+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:07.268281+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2552,"log_dups_size":0,"ondisk_log_size":2552,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20854028,"num_objects":2546,"num_object_clones":0,"num_object_copies":10184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2550,"num_write_kb":20380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2658","reported_seq":2721,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.155317+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:07.155317+0000","last_peered":"2024-09-15T21:04:07.155317+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:07.155317+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2658,"log_dups_size":0,"ondisk_log_size":2658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21730572,"num_objects":2653,"num_object_clones":0,"num_object_copies":10612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":21236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2534","reported_seq":2588,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382738+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:07.382738+0000","last_peered":"2024-09-15T21:04:07.382738+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:07.382738+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2534,"log_dups_size":0,"ondisk_log_size":2534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20687384,"num_objects":2526,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2531,"num_write_kb":20225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":194}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2631","reported_seq":2705,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.953268+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:10.953268+0000","last_peered":"2024-09-15T21:04:10.953268+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:10.953268+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21533964,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2471","reported_seq":2700,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.442186+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:07.442186+0000","last_peered":"2024-09-15T21:04:07.442186+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:07.442186+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2471,"log_dups_size":0,"ondisk_log_size":2471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20197266,"num_objects":2466,"num_object_clones":0,"num_object_copies":9864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":19732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2445","reported_seq":2478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.468081+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:07.468081+0000","last_peered":"2024-09-15T21:04:07.468081+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:07.468081+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2445,"log_dups_size":0,"ondisk_log_size":2445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20019846,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":19551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2569","reported_seq":2643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.954341+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:10.954341+0000","last_peered":"2024-09-15T21:04:10.954341+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:10.954341+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21035654,"num_objects":2568,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2568,"num_write_kb":20543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2473","reported_seq":2524,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.382442+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:07.382442+0000","last_peered":"2024-09-15T21:04:07.382442+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:07.382442+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2473,"log_dups_size":0,"ondisk_log_size":2473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20223244,"num_objects":2469,"num_object_clones":0,"num_object_copies":9876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2471,"num_write_kb":19764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":190}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2576","reported_seq":2653,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.998516+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:10.998516+0000","last_peered":"2024-09-15T21:04:10.998516+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:10.998516+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21075212,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2536","reported_seq":2599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:07.292801+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:07.292801+0000","last_peered":"2024-09-15T21:04:07.292801+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:07.292801+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20731148,"num_objects":2531,"num_object_clones":0,"num_object_copies":10124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2534,"num_write_kb":20260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":331343528,"num_objects":40452,"num_object_clones":0,"num_object_copies":161808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36995,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40479,"num_write_kb":323714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517484544,"data_stored":517484544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40506,"ondisk_log_size":40506,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308300,"kb_used_data":159996,"kb_used_omap":4,"kb_used_meta":148283,"kb_avail":94063540,"statfs":{"total":96636764160,"available":96321064960,"internally_reserved":0,"allocated":163835904,"data_stored":163678954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":151842512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,9,4,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301280,"kb_used_data":165968,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94070560,"statfs":{"total":96636764160,"available":96328253440,"internally_reserved":0,"allocated":169951232,"data_stored":169794282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80832,"kb_used_data":18672,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94291008,"statfs":{"total":96636764160,"available":96553992192,"internally_reserved":0,"allocated":19120128,"data_stored":18963178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63631055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303596,"kb_used_data":159516,"kb_used_omap":4,"kb_used_meta":144059,"kb_avail":94068244,"statfs":{"total":96636764160,"available":96325881856,"internally_reserved":0,"allocated":163344384,"data_stored":163187434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":147517134},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164200448,"data_stored":164200448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18890752,"data_stored":18890752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169938944,"data_stored":169938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":164454400,"data_stored":164454400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:13.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:13.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2452", 'reported_seq': 2490, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.460441+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:07.460441+0000', 'last_peered': '2024-09-15T21:04:07.460441+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:07.460441+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2452, 'log_dups_size': 0, 'ondisk_log_size': 2452, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20086784, 'num_objects': 2452, 'num_object_clones': 0, 'num_object_copies': 9808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2452, 'num_write_kb': 19616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 176}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.622 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:13.622 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2575", 'reported_seq': 2637, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.251738+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:07.251738+0000', 'last_peered': '2024-09-15T21:04:07.251738+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:07.251738+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2575, 'log_dups_size': 0, 'ondisk_log_size': 2575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21058828, 'num_objects': 2571, 'num_object_clones': 0, 'num_object_copies': 10284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2573, 'num_write_kb': 20580, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2504", 'reported_seq': 2577, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:10.950011+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:10.950011+0000', 'last_peered': '2024-09-15T21:04:10.950011+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:10.950011+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2504, 'log_dups_size': 0, 'ondisk_log_size': 2504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20486790, 'num_objects': 2501, 'num_object_clones': 0, 'num_object_copies': 10004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2503, 'num_write_kb': 20014, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 243}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2526", 'reported_seq': 2589, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.207045+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:07.207045+0000', 'last_peered': '2024-09-15T21:04:07.207045+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:07.207045+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2526, 'log_dups_size': 0, 'ondisk_log_size': 2526, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20657420, 'num_objects': 2522, 'num_object_clones': 0, 'num_object_copies': 10088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2524, 'num_write_kb': 20181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2505", 'reported_seq': 2555, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.412894+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:07.412894+0000', 'last_peered': '2024-09-15T21:04:07.412894+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:07.412894+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2505, 'log_dups_size': 0, 'ondisk_log_size': 2505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20493580, 'num_objects': 2502, 'num_object_clones': 0, 'num_object_copies': 10008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2502, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2503, 'num_write_kb': 20021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2499", 'reported_seq': 2538, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.469983+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:07.469983+0000', 'last_peered': '2024-09-15T21:04:07.469983+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:07.469983+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2499, 'log_dups_size': 0, 'ondisk_log_size': 2499, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20471808, 'num_objects': 2499, 'num_object_clones': 0, 'num_object_copies': 9996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2499, 'num_write_kb': 19992, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2552", 'reported_seq': 2616, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.268281+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:07.268281+0000', 'last_peered': '2024-09-15T21:04:07.268281+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:07.268281+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2552, 'log_dups_size': 0, 'ondisk_log_size': 2552, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20854028, 'num_objects': 2546, 'num_object_clones': 0, 'num_object_copies': 10184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2550, 'num_write_kb': 20380, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 180}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:13.623 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2658", 'reported_seq': 2721, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.155317+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:07.155317+0000', 'last_peered': '2024-09-15T21:04:07.155317+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:07.155317+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2658, 'log_dups_size': 0, 'ondisk_log_size': 2658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21730572, 'num_objects': 2653, 'num_object_clones': 0, 'num_object_copies': 10612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2656, 'num_write_kb': 21236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2534", 'reported_seq': 2588, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.382738+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:07.382738+0000', 'last_peered': '2024-09-15T21:04:07.382738+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:07.382738+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2534, 'log_dups_size': 0, 'ondisk_log_size': 2534, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20687384, 'num_objects': 2526, 'num_object_clones': 0, 'num_object_copies': 10104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2526, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2531, 'num_write_kb': 20225, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 194}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2631", 'reported_seq': 2705, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:10.953268+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:10.953268+0000', 'last_peered': '2024-09-15T21:04:10.953268+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:10.953268+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2631, 'log_dups_size': 0, 'ondisk_log_size': 2631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21533964, 'num_objects': 2629, 'num_object_clones': 0, 'num_object_copies': 10516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2629, 'num_write_kb': 21030, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 294}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2471", 'reported_seq': 2700, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.442186+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:07.442186+0000', 'last_peered': '2024-09-15T21:04:07.442186+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:07.442186+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2471, 'log_dups_size': 0, 'ondisk_log_size': 2471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20197266, 'num_objects': 2466, 'num_object_clones': 0, 'num_object_copies': 9864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2172, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2468, 'num_write_kb': 19732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 91, 'num_bytes_recovered': 720896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 294}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2445", 'reported_seq': 2478, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.468081+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:07.468081+0000', 'last_peered': '2024-09-15T21:04:07.468081+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:07.468081+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2445, 'log_dups_size': 0, 'ondisk_log_size': 2445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20019846, 'num_objects': 2444, 'num_object_clones': 0, 'num_object_copies': 9776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2444, 'num_write_kb': 19551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:13.624 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2569", 'reported_seq': 2643, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:10.954341+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:10.954341+0000', 'last_peered': '2024-09-15T21:04:10.954341+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:10.954341+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2569, 'log_dups_size': 0, 'ondisk_log_size': 2569, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21035654, 'num_objects': 2568, 'num_object_clones': 0, 'num_object_copies': 10272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2568, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2568, 'num_write_kb': 20543, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2473", 'reported_seq': 2524, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.382442+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:07.382442+0000', 'last_peered': '2024-09-15T21:04:07.382442+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:07.382442+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2473, 'log_dups_size': 0, 'ondisk_log_size': 2473, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20223244, 'num_objects': 2469, 'num_object_clones': 0, 'num_object_copies': 9876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2471, 'num_write_kb': 19764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 190}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2576", 'reported_seq': 2653, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:10.998516+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:10.998516+0000', 'last_peered': '2024-09-15T21:04:10.998516+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:10.998516+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2576, 'log_dups_size': 0, 'ondisk_log_size': 2576, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21075212, 'num_objects': 2573, 'num_object_clones': 0, 'num_object_copies': 10292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2573, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2574, 'num_write_kb': 20589, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 292}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:13.625 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2536", 'reported_seq': 2599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:07.292801+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:07.292801+0000', 'last_peered': '2024-09-15T21:04:07.292801+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:07.292801+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2536, 'log_dups_size': 0, 'ondisk_log_size': 2536, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20731148, 'num_objects': 2531, 'num_object_clones': 0, 'num_object_copies': 10124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2534, 'num_write_kb': 20260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:13.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:13.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 50 20 42490 42470 6.63503 2.76562 0.0500479 0.0235258 2024-09-15T21:04:14.344 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:14.345 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36937/167748 objects degraded (22.019%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":120,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41937,"data_bytes":343508648,"bytes_used":1024987136,"bytes_avail":385522069504,"bytes_total":386547056640,"degraded_objects":36937,"degraded_total":167748,"degraded_ratio":0.22019338531606933,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99645,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"write_bytes_sec":4350297,"read_op_per_sec":0,"write_op_per_sec":531},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:14.375 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:14.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:14.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 51 20 42846 42826 6.55946 2.78125 0.0271169 0.023798 2024-09-15T21:04:15.016 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:15.017 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:15.051 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-15T21:04:13.920203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":343508648,"num_objects":41937,"num_object_clones":0,"num_object_copies":167748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41964,"num_write_kb":335594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41991,"ondisk_log_size":41991,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1000964,"kb_used_data":519428,"kb_used_omap":14,"kb_used_meta":481457,"kb_avail":376486396,"statfs":{"total":386547056640,"available":385522069504,"internally_reserved":0,"allocated":531894272,"data_stored":531266472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":493012530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,7,17,8,14,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":52215808,"num_objects":6374,"num_object_clones":0,"num_object_copies":25496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6374,"num_write_kb":50992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"1.f","version":"31'2572","reported_seq":2612,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.389209+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:12.389209+0000","last_peered":"2024-09-15T21:04:12.389209+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:12.389209+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2572,"log_dups_size":0,"ondisk_log_size":2572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21069824,"num_objects":2572,"num_object_clones":0,"num_object_copies":10288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2572,"num_write_kb":20576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":296}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2689","reported_seq":2751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.285425+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:12.285425+0000","last_peered":"2024-09-15T21:04:12.285425+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:12.285425+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2689,"log_dups_size":0,"ondisk_log_size":2689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21992716,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":21492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":305}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2504","reported_seq":2577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.950011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:10.950011+0000","last_peered":"2024-09-15T21:04:10.950011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:10.950011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2504,"log_dups_size":0,"ondisk_log_size":2504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20486790,"num_objects":2501,"num_object_clones":0,"num_object_copies":10004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":20014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":243}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2646","reported_seq":2709,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.310027+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:12.310027+0000","last_peered":"2024-09-15T21:04:12.310027+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:12.310027+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2646,"log_dups_size":0,"ondisk_log_size":2646,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21640460,"num_objects":2642,"num_object_clones":0,"num_object_copies":10568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2644,"num_write_kb":21141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2664","reported_seq":2712,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.499564+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:12.499564+0000","last_peered":"2024-09-15T21:04:12.499564+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:12.499564+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796108,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":350}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2631","reported_seq":2668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.459840+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:12.459840+0000","last_peered":"2024-09-15T21:04:12.459840+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:12.459840+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21553152,"num_objects":2631,"num_object_clones":0,"num_object_copies":10524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2631,"num_write_kb":21048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":315}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2667","reported_seq":2730,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.315166+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:12.315166+0000","last_peered":"2024-09-15T21:04:12.315166+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:12.315166+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2667,"log_dups_size":0,"ondisk_log_size":2667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21796108,"num_objects":2661,"num_object_clones":0,"num_object_copies":10644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2665,"num_write_kb":21300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":295}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2761","reported_seq":2824,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.251354+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:12.251354+0000","last_peered":"2024-09-15T21:04:12.251354+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:12.251354+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2761,"log_dups_size":0,"ondisk_log_size":2761,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22574348,"num_objects":2756,"num_object_clones":0,"num_object_copies":11024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2759,"num_write_kb":22060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2645","reported_seq":2699,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.367792+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:12.367792+0000","last_peered":"2024-09-15T21:04:12.367792+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:12.367792+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2645,"log_dups_size":0,"ondisk_log_size":2645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21596696,"num_objects":2637,"num_object_clones":0,"num_object_copies":10548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2642,"num_write_kb":21113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2631","reported_seq":2705,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.953268+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:10.953268+0000","last_peered":"2024-09-15T21:04:10.953268+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:10.953268+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2631,"log_dups_size":0,"ondisk_log_size":2631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21533964,"num_objects":2629,"num_object_clones":0,"num_object_copies":10516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2629,"num_write_kb":21030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":294}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2584","reported_seq":2931,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.451077+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:12.451077+0000","last_peered":"2024-09-15T21:04:12.451077+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:12.451077+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2584,"log_dups_size":0,"ondisk_log_size":2584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21122962,"num_objects":2579,"num_object_clones":0,"num_object_copies":10316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2581,"num_write_kb":20636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":465}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2568","reported_seq":2602,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.434582+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:12.434582+0000","last_peered":"2024-09-15T21:04:12.434582+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:12.434582+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2568,"log_dups_size":0,"ondisk_log_size":2568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21027462,"num_objects":2567,"num_object_clones":0,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":20535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":311}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2569","reported_seq":2643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.954341+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:10.954341+0000","last_peered":"2024-09-15T21:04:10.954341+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:10.954341+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21035654,"num_objects":2568,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2568,"num_write_kb":20543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":282}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2620","reported_seq":2669,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.431710+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:12.431710+0000","last_peered":"2024-09-15T21:04:12.431710+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:12.431710+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2620,"log_dups_size":0,"ondisk_log_size":2620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21427468,"num_objects":2616,"num_object_clones":0,"num_object_copies":10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":20940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2576","reported_seq":2653,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:10.998516+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:10.998516+0000","last_peered":"2024-09-15T21:04:10.998516+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:10.998516+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2576,"log_dups_size":0,"ondisk_log_size":2576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21075212,"num_objects":2573,"num_object_clones":0,"num_object_copies":10292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2574,"num_write_kb":20589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":292}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2664","reported_seq":2727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:12.362402+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:12.362402+0000","last_peered":"2024-09-15T21:04:12.362402+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:12.362402+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2664,"log_dups_size":0,"ondisk_log_size":2664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21779724,"num_objects":2659,"num_object_clones":0,"num_object_copies":10636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":21284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":344}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":343508648,"num_objects":41937,"num_object_clones":0,"num_object_copies":167748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36937,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41964,"num_write_kb":335594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":533426176,"data_stored":533426176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41991,"ondisk_log_size":41991,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":303484,"kb_used_data":167660,"kb_used_omap":4,"kb_used_meta":135803,"kb_avail":94068356,"statfs":{"total":96636764160,"available":96325996544,"internally_reserved":0,"allocated":171683840,"data_stored":171526890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":139062992},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,3,4,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301280,"kb_used_data":165968,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":94070560,"statfs":{"total":96636764160,"available":96328253440,"internally_reserved":0,"allocated":169951232,"data_stored":169794282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051588,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80832,"kb_used_data":18672,"kb_used_omap":4,"kb_used_meta":62139,"kb_avail":94291008,"statfs":{"total":96636764160,"available":96553992192,"internally_reserved":0,"allocated":19120128,"data_stored":18963178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":63631055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315368,"kb_used_data":167128,"kb_used_omap":4,"kb_used_meta":148219,"kb_avail":94056472,"statfs":{"total":96636764160,"available":96313827328,"internally_reserved":0,"allocated":171139072,"data_stored":170982122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":151776974},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172199936,"data_stored":172199936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":18890752,"data_stored":18890752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":169938944,"data_stored":169938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172396544,"data_stored":172396544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:15.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 52 20 43237 43217 6.49206 3.05469 0.0484138 0.0240504 2024-09-15T21:04:16.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 53 20 43637 43617 6.42853 3.125 0.0388224 0.0242923 2024-09-15T21:04:17.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 54 20 44042 44022 6.36808 3.16406 0.0364028 0.0245228 2024-09-15T21:04:18.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:18.633 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 55 20 44413 44393 6.30499 2.89844 0.0472812 0.0247589 2024-09-15T21:04:18.658 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:18.658 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:18.694 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-15T21:04:17.921190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1014800,"kb_used_data":549904,"kb_used_omap":14,"kb_used_meta":464817,"kb_avail":376472560,"statfs":{"total":386547056640,"available":385507901440,"internally_reserved":0,"allocated":563101696,"data_stored":562473896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":475973170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,8,4,11,11,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48168960,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002870"},"pg_stats":[{"pgid":"1.f","version":"31'2694","reported_seq":2734,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.453567+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:17.453567+0000","last_peered":"2024-09-15T21:04:17.453567+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:17.453567+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2694,"log_dups_size":0,"ondisk_log_size":2694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22069248,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2694,"num_write_kb":21552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2835","reported_seq":2897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.247260+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:17.247260+0000","last_peered":"2024-09-15T21:04:17.247260+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:17.247260+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2625","reported_seq":2697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.994673+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:15.994673+0000","last_peered":"2024-09-15T21:04:15.994673+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:15.994673+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21478022,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":364}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.295454+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:17.295454+0000","last_peered":"2024-09-15T21:04:17.295454+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:17.295454+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664460,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2788","reported_seq":2838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.370832+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:17.370832+0000","last_peered":"2024-09-15T21:04:17.370832+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:17.370832+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22811916,"num_objects":2785,"num_object_clones":0,"num_object_copies":11140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2748","reported_seq":2787,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.448924+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:17.448924+0000","last_peered":"2024-09-15T21:04:17.448924+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:17.448924+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22511616,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2748,"num_write_kb":21984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2770","reported_seq":2833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.363044+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:17.363044+0000","last_peered":"2024-09-15T21:04:17.363044+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:17.363044+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22639884,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2881","reported_seq":2944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.257527+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:17.257527+0000","last_peered":"2024-09-15T21:04:17.257527+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:17.257527+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23557388,"num_objects":2876,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2773","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.440377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:17.440377+0000","last_peered":"2024-09-15T21:04:17.440377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:17.440377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22645272,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2749","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.860953+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:15.860953+0000","last_peered":"2024-09-15T21:04:15.860953+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:15.860953+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22500620,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2680","reported_seq":3141,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.485170+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:17.485170+0000","last_peered":"2024-09-15T21:04:17.485170+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:17.485170+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2680,"log_dups_size":0,"ondisk_log_size":2680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909394,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":21404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2698","reported_seq":2731,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.512280+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:17.512280+0000","last_peered":"2024-09-15T21:04:17.512280+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:17.512280+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22092422,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2697,"num_write_kb":21575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2696","reported_seq":2770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.846552+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:15.846552+0000","last_peered":"2024-09-15T21:04:15.846552+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:15.846552+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22076038,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":21559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2724","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.369702+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:17.369702+0000","last_peered":"2024-09-15T21:04:17.369702+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:17.369702+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279436,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2698","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.939570+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:15.939570+0000","last_peered":"2024-09-15T21:04:15.939570+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:15.939570+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22074636,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2767","reported_seq":2830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.328150+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:17.328150+0000","last_peered":"2024-09-15T21:04:17.328150+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:17.328150+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623500,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564875264,"data_stored":564875264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314964,"kb_used_data":174980,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94056876,"statfs":{"total":96636764160,"available":96314241024,"internally_reserved":0,"allocated":179179520,"data_stored":179022570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313064,"kb_used_data":173592,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94058776,"statfs":{"total":96636764160,"available":96316186624,"internally_reserved":0,"allocated":177758208,"data_stored":177601258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76172,"kb_used_data":26492,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94295668,"statfs":{"total":96636764160,"available":96558764032,"internally_reserved":0,"allocated":27127808,"data_stored":26970858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310600,"kb_used_data":174840,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061240,"statfs":{"total":96636764160,"available":96318709760,"internally_reserved":0,"allocated":179036160,"data_stored":178879210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179945472,"data_stored":179945472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26984448,"data_stored":26984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177737728,"data_stored":177737728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180207616,"data_stored":180207616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:18.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:19.256 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:19.257 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:19.286 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-15T21:04:17.921190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1014800,"kb_used_data":549904,"kb_used_omap":14,"kb_used_meta":464817,"kb_avail":376472560,"statfs":{"total":386547056640,"available":385507901440,"internally_reserved":0,"allocated":563101696,"data_stored":562473896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":475973170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,8,4,11,11,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":48168960,"num_objects":5880,"num_object_clones":0,"num_object_copies":23520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5880,"num_write_kb":47040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002870"},"pg_stats":[{"pgid":"1.f","version":"31'2694","reported_seq":2734,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.453567+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:17.453567+0000","last_peered":"2024-09-15T21:04:17.453567+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:17.453567+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2694,"log_dups_size":0,"ondisk_log_size":2694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22069248,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2694,"num_write_kb":21552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2835","reported_seq":2897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.247260+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:17.247260+0000","last_peered":"2024-09-15T21:04:17.247260+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:17.247260+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2625","reported_seq":2697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.994673+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:15.994673+0000","last_peered":"2024-09-15T21:04:15.994673+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:15.994673+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21478022,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":364}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.295454+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:17.295454+0000","last_peered":"2024-09-15T21:04:17.295454+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:17.295454+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664460,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2788","reported_seq":2838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.370832+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:17.370832+0000","last_peered":"2024-09-15T21:04:17.370832+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:17.370832+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22811916,"num_objects":2785,"num_object_clones":0,"num_object_copies":11140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2748","reported_seq":2787,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.448924+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:17.448924+0000","last_peered":"2024-09-15T21:04:17.448924+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:17.448924+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22511616,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2748,"num_write_kb":21984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2770","reported_seq":2833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.363044+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:17.363044+0000","last_peered":"2024-09-15T21:04:17.363044+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:17.363044+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22639884,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2881","reported_seq":2944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.257527+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:17.257527+0000","last_peered":"2024-09-15T21:04:17.257527+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:17.257527+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23557388,"num_objects":2876,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2773","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.440377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:17.440377+0000","last_peered":"2024-09-15T21:04:17.440377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:17.440377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22645272,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2749","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.860953+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:15.860953+0000","last_peered":"2024-09-15T21:04:15.860953+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:15.860953+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22500620,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2680","reported_seq":3141,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.485170+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:17.485170+0000","last_peered":"2024-09-15T21:04:17.485170+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:17.485170+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2680,"log_dups_size":0,"ondisk_log_size":2680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909394,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":21404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2698","reported_seq":2731,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.512280+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:17.512280+0000","last_peered":"2024-09-15T21:04:17.512280+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:17.512280+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22092422,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2697,"num_write_kb":21575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2696","reported_seq":2770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.846552+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:15.846552+0000","last_peered":"2024-09-15T21:04:15.846552+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:15.846552+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22076038,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":21559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2724","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.369702+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:17.369702+0000","last_peered":"2024-09-15T21:04:17.369702+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:17.369702+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279436,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2698","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.939570+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:15.939570+0000","last_peered":"2024-09-15T21:04:15.939570+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:15.939570+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22074636,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2767","reported_seq":2830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.328150+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:17.328150+0000","last_peered":"2024-09-15T21:04:17.328150+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:17.328150+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623500,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564875264,"data_stored":564875264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314964,"kb_used_data":174980,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94056876,"statfs":{"total":96636764160,"available":96314241024,"internally_reserved":0,"allocated":179179520,"data_stored":179022570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313064,"kb_used_data":173592,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94058776,"statfs":{"total":96636764160,"available":96316186624,"internally_reserved":0,"allocated":177758208,"data_stored":177601258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76172,"kb_used_data":26492,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94295668,"statfs":{"total":96636764160,"available":96558764032,"internally_reserved":0,"allocated":27127808,"data_stored":26970858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310600,"kb_used_data":174840,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061240,"statfs":{"total":96636764160,"available":96318709760,"internally_reserved":0,"allocated":179036160,"data_stored":178879210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179945472,"data_stored":179945472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26984448,"data_stored":26984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177737728,"data_stored":177737728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180207616,"data_stored":180207616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:19.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:19.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2694", 'reported_seq': 2734, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.453567+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:17.453567+0000', 'last_peered': '2024-09-15T21:04:17.453567+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:17.453567+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2694, 'log_dups_size': 0, 'ondisk_log_size': 2694, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22069248, 'num_objects': 2694, 'num_object_clones': 0, 'num_object_copies': 10776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2694, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2694, 'num_write_kb': 21552, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 418}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:19.287 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2835", 'reported_seq': 2897, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.247260+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:17.247260+0000', 'last_peered': '2024-09-15T21:04:17.247260+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:17.247260+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2835, 'log_dups_size': 0, 'ondisk_log_size': 2835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23188748, 'num_objects': 2831, 'num_object_clones': 0, 'num_object_copies': 11324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2833, 'num_write_kb': 22660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 451}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:19.287 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2625", 'reported_seq': 2697, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:15.994673+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:15.994673+0000', 'last_peered': '2024-09-15T21:04:15.994673+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:15.994673+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2625, 'log_dups_size': 0, 'ondisk_log_size': 2625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21478022, 'num_objects': 2622, 'num_object_clones': 0, 'num_object_copies': 10488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2624, 'num_write_kb': 20982, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 364}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2771", 'reported_seq': 2834, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.295454+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:17.295454+0000', 'last_peered': '2024-09-15T21:04:17.295454+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:17.295454+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2771, 'log_dups_size': 0, 'ondisk_log_size': 2771, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22664460, 'num_objects': 2767, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2769, 'num_write_kb': 22141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 433}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2788", 'reported_seq': 2838, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.370832+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:17.370832+0000', 'last_peered': '2024-09-15T21:04:17.370832+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:17.370832+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2788, 'log_dups_size': 0, 'ondisk_log_size': 2788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22811916, 'num_objects': 2785, 'num_object_clones': 0, 'num_object_copies': 11140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2786, 'num_write_kb': 22285, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2748", 'reported_seq': 2787, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.448924+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:17.448924+0000', 'last_peered': '2024-09-15T21:04:17.448924+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:17.448924+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2748, 'log_dups_size': 0, 'ondisk_log_size': 2748, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22511616, 'num_objects': 2748, 'num_object_clones': 0, 'num_object_copies': 10992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2748, 'num_write_kb': 21984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 432}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2770", 'reported_seq': 2833, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.363044+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:17.363044+0000', 'last_peered': '2024-09-15T21:04:17.363044+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:17.363044+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2770, 'log_dups_size': 0, 'ondisk_log_size': 2770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22639884, 'num_objects': 2764, 'num_object_clones': 0, 'num_object_copies': 11056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2764, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2768, 'num_write_kb': 22124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 398}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2881", 'reported_seq': 2944, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.257527+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:17.257527+0000', 'last_peered': '2024-09-15T21:04:17.257527+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:17.257527+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2881, 'log_dups_size': 0, 'ondisk_log_size': 2881, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23557388, 'num_objects': 2876, 'num_object_clones': 0, 'num_object_copies': 11504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2879, 'num_write_kb': 23020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2773", 'reported_seq': 2827, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.440377+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:17.440377+0000', 'last_peered': '2024-09-15T21:04:17.440377+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:17.440377+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2773, 'log_dups_size': 0, 'ondisk_log_size': 2773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22645272, 'num_objects': 2765, 'num_object_clones': 0, 'num_object_copies': 11060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2765, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2770, 'num_write_kb': 22137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 433}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2749", 'reported_seq': 2823, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:15.860953+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:15.860953+0000', 'last_peered': '2024-09-15T21:04:15.860953+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:15.860953+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2749, 'log_dups_size': 0, 'ondisk_log_size': 2749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22500620, 'num_objects': 2747, 'num_object_clones': 0, 'num_object_copies': 10988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2747, 'num_write_kb': 21974, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 412}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2680", 'reported_seq': 3141, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.485170+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:17.485170+0000', 'last_peered': '2024-09-15T21:04:17.485170+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:17.485170+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2680, 'log_dups_size': 0, 'ondisk_log_size': 2680, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21909394, 'num_objects': 2675, 'num_object_clones': 0, 'num_object_copies': 10700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2057, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2677, 'num_write_kb': 21404, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 206, 'num_bytes_recovered': 1662976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 618}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2698", 'reported_seq': 2731, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.512280+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:17.512280+0000', 'last_peered': '2024-09-15T21:04:17.512280+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:17.512280+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2698, 'log_dups_size': 0, 'ondisk_log_size': 2698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22092422, 'num_objects': 2697, 'num_object_clones': 0, 'num_object_copies': 10788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2697, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2697, 'num_write_kb': 21575, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 441}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:19.289 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2696", 'reported_seq': 2770, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:15.846552+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:15.846552+0000', 'last_peered': '2024-09-15T21:04:15.846552+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:15.846552+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2696, 'log_dups_size': 0, 'ondisk_log_size': 2696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22076038, 'num_objects': 2695, 'num_object_clones': 0, 'num_object_copies': 10780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2695, 'num_write_kb': 21559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2724", 'reported_seq': 2775, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.369702+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:17.369702+0000', 'last_peered': '2024-09-15T21:04:17.369702+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:17.369702+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2724, 'log_dups_size': 0, 'ondisk_log_size': 2724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22279436, 'num_objects': 2720, 'num_object_clones': 0, 'num_object_copies': 10880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2722, 'num_write_kb': 21772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 441}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2698", 'reported_seq': 2775, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:15.939570+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:15.939570+0000', 'last_peered': '2024-09-15T21:04:15.939570+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:15.939570+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2698, 'log_dups_size': 0, 'ondisk_log_size': 2698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22074636, 'num_objects': 2695, 'num_object_clones': 0, 'num_object_copies': 10780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2696, 'num_write_kb': 21565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 414}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:19.290 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2767", 'reported_seq': 2830, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:17.328150+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:17.328150+0000', 'last_peered': '2024-09-15T21:04:17.328150+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:17.328150+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2767, 'log_dups_size': 0, 'ondisk_log_size': 2767, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22623500, 'num_objects': 2762, 'num_object_clones': 0, 'num_object_copies': 11048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2765, 'num_write_kb': 22108, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 447}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:19.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:19.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 56 20 44779 44759 6.24346 2.85938 0.0458557 0.0250089 2024-09-15T21:04:20.041 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:20.042 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36880/175372 objects degraded (21.030%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":125,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43843,"data_bytes":359122600,"bytes_used":1039155200,"bytes_avail":385507901440,"bytes_total":386547056640,"degraded_objects":36880,"degraded_total":175372,"degraded_ratio":0.21029582829642129,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":121485,"recovering_keys_per_sec":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"write_bytes_sec":4013120,"read_op_per_sec":0,"write_op_per_sec":489},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:20.070 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:20.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:20.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 57 20 45127 45107 6.18162 2.71875 0.0601558 0.0252592 2024-09-15T21:04:20.689 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:20.689 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:20.717 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-09-15T21:04:19.921722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1014800,"kb_used_data":549904,"kb_used_omap":14,"kb_used_meta":464817,"kb_avail":376472560,"statfs":{"total":386547056640,"available":385507901440,"internally_reserved":0,"allocated":563101696,"data_stored":562473896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":475973170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,8,4,11,11,9,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31563776,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3853,"num_write_kb":30824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":942080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002967"},"pg_stats":[{"pgid":"1.f","version":"31'2694","reported_seq":2734,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.453567+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:17.453567+0000","last_peered":"2024-09-15T21:04:17.453567+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:17.453567+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2694,"log_dups_size":0,"ondisk_log_size":2694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22069248,"num_objects":2694,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2694,"num_write_kb":21552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":418}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2835","reported_seq":2897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.247260+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:17.247260+0000","last_peered":"2024-09-15T21:04:17.247260+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:17.247260+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2835,"log_dups_size":0,"ondisk_log_size":2835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23188748,"num_objects":2831,"num_object_clones":0,"num_object_copies":11324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2833,"num_write_kb":22660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":451}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2625","reported_seq":2697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.994673+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:15.994673+0000","last_peered":"2024-09-15T21:04:15.994673+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:15.994673+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2625,"log_dups_size":0,"ondisk_log_size":2625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21478022,"num_objects":2622,"num_object_clones":0,"num_object_copies":10488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2624,"num_write_kb":20982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":364}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2771","reported_seq":2834,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.295454+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:17.295454+0000","last_peered":"2024-09-15T21:04:17.295454+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:17.295454+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2771,"log_dups_size":0,"ondisk_log_size":2771,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664460,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2769,"num_write_kb":22141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":433}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2788","reported_seq":2838,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.370832+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:17.370832+0000","last_peered":"2024-09-15T21:04:17.370832+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:17.370832+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2788,"log_dups_size":0,"ondisk_log_size":2788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22811916,"num_objects":2785,"num_object_clones":0,"num_object_copies":11140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":22285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2748","reported_seq":2787,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.448924+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:17.448924+0000","last_peered":"2024-09-15T21:04:17.448924+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:17.448924+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2748,"log_dups_size":0,"ondisk_log_size":2748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22511616,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2748,"num_write_kb":21984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2770","reported_seq":2833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.363044+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:17.363044+0000","last_peered":"2024-09-15T21:04:17.363044+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:17.363044+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22639884,"num_objects":2764,"num_object_clones":0,"num_object_copies":11056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":22124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2881","reported_seq":2944,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.257527+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:17.257527+0000","last_peered":"2024-09-15T21:04:17.257527+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:17.257527+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2881,"log_dups_size":0,"ondisk_log_size":2881,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23557388,"num_objects":2876,"num_object_clones":0,"num_object_copies":11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2879,"num_write_kb":23020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2773","reported_seq":2827,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.440377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:17.440377+0000","last_peered":"2024-09-15T21:04:17.440377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:17.440377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2773,"log_dups_size":0,"ondisk_log_size":2773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22645272,"num_objects":2765,"num_object_clones":0,"num_object_copies":11060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":22137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":433}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2749","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.860953+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:15.860953+0000","last_peered":"2024-09-15T21:04:15.860953+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:15.860953+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2749,"log_dups_size":0,"ondisk_log_size":2749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22500620,"num_objects":2747,"num_object_clones":0,"num_object_copies":10988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2747,"num_write_kb":21974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2680","reported_seq":3141,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.485170+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:17.485170+0000","last_peered":"2024-09-15T21:04:17.485170+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:17.485170+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2680,"log_dups_size":0,"ondisk_log_size":2680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21909394,"num_objects":2675,"num_object_clones":0,"num_object_copies":10700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2677,"num_write_kb":21404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2698","reported_seq":2731,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.512280+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:17.512280+0000","last_peered":"2024-09-15T21:04:17.512280+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:17.512280+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22092422,"num_objects":2697,"num_object_clones":0,"num_object_copies":10788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2697,"num_write_kb":21575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2696","reported_seq":2770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.846552+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:15.846552+0000","last_peered":"2024-09-15T21:04:15.846552+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:15.846552+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2696,"log_dups_size":0,"ondisk_log_size":2696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22076038,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2695,"num_write_kb":21559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":409}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2724","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.369702+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:17.369702+0000","last_peered":"2024-09-15T21:04:17.369702+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:17.369702+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2724,"log_dups_size":0,"ondisk_log_size":2724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279436,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2722,"num_write_kb":21772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":441}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2698","reported_seq":2775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:15.939570+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:15.939570+0000","last_peered":"2024-09-15T21:04:15.939570+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:15.939570+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2698,"log_dups_size":0,"ondisk_log_size":2698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22074636,"num_objects":2695,"num_object_clones":0,"num_object_copies":10780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2696,"num_write_kb":21565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":414}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2767","reported_seq":2830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:17.328150+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:17.328150+0000","last_peered":"2024-09-15T21:04:17.328150+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:17.328150+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2767,"log_dups_size":0,"ondisk_log_size":2767,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22623500,"num_objects":2762,"num_object_clones":0,"num_object_copies":11048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2765,"num_write_kb":22108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":447}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359122600,"num_objects":43843,"num_object_clones":0,"num_object_copies":175372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43870,"num_write_kb":350842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564875264,"data_stored":564875264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43897,"ondisk_log_size":43897,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314964,"kb_used_data":174980,"kb_used_omap":4,"kb_used_meta":139963,"kb_avail":94056876,"statfs":{"total":96636764160,"available":96314241024,"internally_reserved":0,"allocated":179179520,"data_stored":179022570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":143322832},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313064,"kb_used_data":173592,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":94058776,"statfs":{"total":96636764160,"available":96316186624,"internally_reserved":0,"allocated":177758208,"data_stored":177601258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051589,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76172,"kb_used_data":26492,"kb_used_omap":4,"kb_used_meta":49659,"kb_avail":94295668,"statfs":{"total":96636764160,"available":96558764032,"internally_reserved":0,"allocated":27127808,"data_stored":26970858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":50851535},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310600,"kb_used_data":174840,"kb_used_omap":4,"kb_used_meta":135739,"kb_avail":94061240,"statfs":{"total":96636764160,"available":96318709760,"internally_reserved":0,"allocated":179036160,"data_stored":178879210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":138997454},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179945472,"data_stored":179945472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26984448,"data_stored":26984448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177737728,"data_stored":177737728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":180207616,"data_stored":180207616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:21.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 58 20 45498 45478 6.125 2.89844 0.0561624 0.0254928 2024-09-15T21:04:22.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 59 20 45874 45854 6.07097 2.9375 0.054831 0.0257241 2024-09-15T21:04:23.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:04:23.632319+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0259334 2024-09-15T21:04:23.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:04:23.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 60 20 46274 46254 6.02187 3.125 0.0645565 0.0259334 2024-09-15T21:04:23.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:24.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:24.329 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:24.355 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-15T21:04:23.922660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082724,"kb_used_data":578980,"kb_used_omap":14,"kb_used_meta":503665,"kb_avail":376404636,"statfs":{"total":386547056640,"available":385438347264,"internally_reserved":0,"allocated":592875520,"data_stored":592247720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":515753522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,8,20,16,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42860544,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5232,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002941"},"pg_stats":[{"pgid":"1.f","version":"31'2783","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.524949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:22.524949+0000","last_peered":"2024-09-15T21:04:22.524949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:22.524949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2783,"log_dups_size":0,"ondisk_log_size":2783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22798336,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2942","reported_seq":3004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.384814+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:22.384814+0000","last_peered":"2024-09-15T21:04:22.384814+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:22.384814+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24065292,"num_objects":2938,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2940,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2752","reported_seq":2824,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.941761+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:20.941761+0000","last_peered":"2024-09-15T21:04:20.941761+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:20.941761+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2752,"log_dups_size":0,"ondisk_log_size":2752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518406,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":21998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2887","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.332801+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:22.332801+0000","last_peered":"2024-09-15T21:04:22.332801+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:22.332801+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2887,"log_dups_size":0,"ondisk_log_size":2887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614732,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":23069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2900","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.515394+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:22.515394+0000","last_peered":"2024-09-15T21:04:22.515394+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:22.515394+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2900,"log_dups_size":0,"ondisk_log_size":2900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23729420,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2898,"num_write_kb":23181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2865","reported_seq":2902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.543672+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:22.543672+0000","last_peered":"2024-09-15T21:04:22.543672+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:22.543672+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2885","reported_seq":2947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.351421+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:22.351421+0000","last_peered":"2024-09-15T21:04:22.351421+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:22.351421+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2989","reported_seq":3052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.289033+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:22.289033+0000","last_peered":"2024-09-15T21:04:22.289033+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:22.289033+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2989,"log_dups_size":0,"ondisk_log_size":2989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24442124,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2890","reported_seq":2943,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.540684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:22.540684+0000","last_peered":"2024-09-15T21:04:22.540684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:22.540684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2890,"log_dups_size":0,"ondisk_log_size":2890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23603736,"num_objects":2882,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":23073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2876","reported_seq":2951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.960661+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:20.960661+0000","last_peered":"2024-09-15T21:04:20.960661+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:20.960661+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2793","reported_seq":3380,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.497447+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:22.497447+0000","last_peered":"2024-09-15T21:04:22.497447+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:22.497447+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22835090,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":22308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2815","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.480279+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:22.480279+0000","last_peered":"2024-09-15T21:04:22.480279+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:22.480279+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050886,"num_objects":2814,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2810","reported_seq":2884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.939052+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:20.939052+0000","last_peered":"2024-09-15T21:04:20.939052+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:20.939052+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23009926,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2809,"num_write_kb":22471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2853","reported_seq":2904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.511171+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:22.511171+0000","last_peered":"2024-09-15T21:04:22.511171+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:22.511171+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336204,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2816","reported_seq":2893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.942178+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:20.942178+0000","last_peered":"2024-09-15T21:04:20.942178+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:20.942178+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041292,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2882","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.234117+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:22.234117+0000","last_peered":"2024-09-15T21:04:22.234117+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:22.234117+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23565580,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595521536,"data_stored":595521536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326716,"kb_used_data":182572,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94045124,"statfs":{"total":96636764160,"available":96302206976,"internally_reserved":0,"allocated":186953728,"data_stored":186796778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,9,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328876,"kb_used_data":181084,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94042964,"statfs":{"total":96636764160,"available":96299995136,"internally_reserved":0,"allocated":185430016,"data_stored":185273066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,10,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104772,"kb_used_data":32884,"kb_used_omap":4,"kb_used_meta":71867,"kb_avail":94267068,"statfs":{"total":96636764160,"available":96529477632,"internally_reserved":0,"allocated":33673216,"data_stored":33516266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73592527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322360,"kb_used_data":182440,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049480,"statfs":{"total":96636764160,"available":96306667520,"internally_reserved":0,"allocated":186818560,"data_stored":186661610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187510784,"data_stored":187510784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34885632,"data_stored":34885632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185352192,"data_stored":185352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187772928,"data_stored":187772928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:24.357 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:24.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 61 20 46625 46605 5.9681 2.74219 0.0278096 0.0261633 2024-09-15T21:04:24.931 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:24.931 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:24.955 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-09-15T21:04:23.922660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082724,"kb_used_data":578980,"kb_used_omap":14,"kb_used_meta":503665,"kb_avail":376404636,"statfs":{"total":386547056640,"available":385438347264,"internally_reserved":0,"allocated":592875520,"data_stored":592247720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":515753522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,8,20,16,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42860544,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5232,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002941"},"pg_stats":[{"pgid":"1.f","version":"31'2783","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.524949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:22.524949+0000","last_peered":"2024-09-15T21:04:22.524949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:22.524949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2783,"log_dups_size":0,"ondisk_log_size":2783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22798336,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2942","reported_seq":3004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.384814+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:22.384814+0000","last_peered":"2024-09-15T21:04:22.384814+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:22.384814+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24065292,"num_objects":2938,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2940,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2752","reported_seq":2824,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.941761+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:20.941761+0000","last_peered":"2024-09-15T21:04:20.941761+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:20.941761+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2752,"log_dups_size":0,"ondisk_log_size":2752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518406,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":21998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2887","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.332801+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:22.332801+0000","last_peered":"2024-09-15T21:04:22.332801+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:22.332801+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2887,"log_dups_size":0,"ondisk_log_size":2887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614732,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":23069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2900","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.515394+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:22.515394+0000","last_peered":"2024-09-15T21:04:22.515394+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:22.515394+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2900,"log_dups_size":0,"ondisk_log_size":2900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23729420,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2898,"num_write_kb":23181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2865","reported_seq":2902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.543672+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:22.543672+0000","last_peered":"2024-09-15T21:04:22.543672+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:22.543672+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2885","reported_seq":2947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.351421+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:22.351421+0000","last_peered":"2024-09-15T21:04:22.351421+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:22.351421+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2989","reported_seq":3052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.289033+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:22.289033+0000","last_peered":"2024-09-15T21:04:22.289033+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:22.289033+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2989,"log_dups_size":0,"ondisk_log_size":2989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24442124,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2890","reported_seq":2943,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.540684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:22.540684+0000","last_peered":"2024-09-15T21:04:22.540684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:22.540684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2890,"log_dups_size":0,"ondisk_log_size":2890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23603736,"num_objects":2882,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":23073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2876","reported_seq":2951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.960661+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:20.960661+0000","last_peered":"2024-09-15T21:04:20.960661+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:20.960661+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2793","reported_seq":3380,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.497447+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:22.497447+0000","last_peered":"2024-09-15T21:04:22.497447+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:22.497447+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22835090,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":22308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2815","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.480279+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:22.480279+0000","last_peered":"2024-09-15T21:04:22.480279+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:22.480279+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050886,"num_objects":2814,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2810","reported_seq":2884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.939052+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:20.939052+0000","last_peered":"2024-09-15T21:04:20.939052+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:20.939052+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23009926,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2809,"num_write_kb":22471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2853","reported_seq":2904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.511171+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:22.511171+0000","last_peered":"2024-09-15T21:04:22.511171+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:22.511171+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336204,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2816","reported_seq":2893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.942178+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:20.942178+0000","last_peered":"2024-09-15T21:04:20.942178+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:20.942178+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041292,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2882","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.234117+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:22.234117+0000","last_peered":"2024-09-15T21:04:22.234117+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:22.234117+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23565580,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595521536,"data_stored":595521536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326716,"kb_used_data":182572,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94045124,"statfs":{"total":96636764160,"available":96302206976,"internally_reserved":0,"allocated":186953728,"data_stored":186796778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,9,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328876,"kb_used_data":181084,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94042964,"statfs":{"total":96636764160,"available":96299995136,"internally_reserved":0,"allocated":185430016,"data_stored":185273066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,10,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104772,"kb_used_data":32884,"kb_used_omap":4,"kb_used_meta":71867,"kb_avail":94267068,"statfs":{"total":96636764160,"available":96529477632,"internally_reserved":0,"allocated":33673216,"data_stored":33516266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73592527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322360,"kb_used_data":182440,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049480,"statfs":{"total":96636764160,"available":96306667520,"internally_reserved":0,"allocated":186818560,"data_stored":186661610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187510784,"data_stored":187510784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34885632,"data_stored":34885632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185352192,"data_stored":185352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187772928,"data_stored":187772928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2783", 'reported_seq': 2823, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.524949+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:22.524949+0000', 'last_peered': '2024-09-15T21:04:22.524949+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:22.524949+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2783, 'log_dups_size': 0, 'ondisk_log_size': 2783, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22798336, 'num_objects': 2783, 'num_object_clones': 0, 'num_object_copies': 11132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2783, 'num_write_kb': 22264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'2942", 'reported_seq': 3004, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.384814+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:22.384814+0000', 'last_peered': '2024-09-15T21:04:22.384814+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:22.384814+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2942, 'log_dups_size': 0, 'ondisk_log_size': 2942, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24065292, 'num_objects': 2938, 'num_object_clones': 0, 'num_object_copies': 11752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2940, 'num_write_kb': 23516, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2752", 'reported_seq': 2824, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:20.941761+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:20.941761+0000', 'last_peered': '2024-09-15T21:04:20.941761+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:20.941761+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2752, 'log_dups_size': 0, 'ondisk_log_size': 2752, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22518406, 'num_objects': 2749, 'num_object_clones': 0, 'num_object_copies': 10996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2751, 'num_write_kb': 21998, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 491}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'2887", 'reported_seq': 2950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.332801+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:22.332801+0000', 'last_peered': '2024-09-15T21:04:22.332801+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:22.332801+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2887, 'log_dups_size': 0, 'ondisk_log_size': 2887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23614732, 'num_objects': 2883, 'num_object_clones': 0, 'num_object_copies': 11532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2885, 'num_write_kb': 23069, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 549}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:24.957 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'2900", 'reported_seq': 2950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.515394+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:22.515394+0000', 'last_peered': '2024-09-15T21:04:22.515394+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:22.515394+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2900, 'log_dups_size': 0, 'ondisk_log_size': 2900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23729420, 'num_objects': 2897, 'num_object_clones': 0, 'num_object_copies': 11588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2898, 'num_write_kb': 23181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 586}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2865", 'reported_seq': 2902, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.543672+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:22.543672+0000', 'last_peered': '2024-09-15T21:04:22.543672+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:22.543672+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2865, 'log_dups_size': 0, 'ondisk_log_size': 2865, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23470080, 'num_objects': 2865, 'num_object_clones': 0, 'num_object_copies': 11460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2865, 'num_write_kb': 22920, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'2885", 'reported_seq': 2947, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.351421+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:22.351421+0000', 'last_peered': '2024-09-15T21:04:22.351421+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:22.351421+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2885, 'log_dups_size': 0, 'ondisk_log_size': 2885, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23581964, 'num_objects': 2879, 'num_object_clones': 0, 'num_object_copies': 11516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2883, 'num_write_kb': 23044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 513}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'2989", 'reported_seq': 3052, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.289033+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:22.289033+0000', 'last_peered': '2024-09-15T21:04:22.289033+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:22.289033+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2989, 'log_dups_size': 0, 'ondisk_log_size': 2989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24442124, 'num_objects': 2984, 'num_object_clones': 0, 'num_object_copies': 11936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2987, 'num_write_kb': 23884, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2890", 'reported_seq': 2943, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.540684+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:22.540684+0000', 'last_peered': '2024-09-15T21:04:22.540684+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:22.540684+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2890, 'log_dups_size': 0, 'ondisk_log_size': 2890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23603736, 'num_objects': 2882, 'num_object_clones': 0, 'num_object_copies': 11528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2887, 'num_write_kb': 23073, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 550}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:24.958 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2876", 'reported_seq': 2951, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:20.960661+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:20.960661+0000', 'last_peered': '2024-09-15T21:04:20.960661+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:20.960661+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2876, 'log_dups_size': 0, 'ondisk_log_size': 2876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23541004, 'num_objects': 2874, 'num_object_clones': 0, 'num_object_copies': 11496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2874, 'num_write_kb': 22990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 539}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2793", 'reported_seq': 3380, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.497447+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:22.497447+0000', 'last_peered': '2024-09-15T21:04:22.497447+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:22.497447+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2793, 'log_dups_size': 0, 'ondisk_log_size': 2793, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22835090, 'num_objects': 2788, 'num_object_clones': 0, 'num_object_copies': 11152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1994, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2790, 'num_write_kb': 22308, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 269, 'num_bytes_recovered': 2179072, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 794}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2815", 'reported_seq': 2849, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.480279+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:22.480279+0000', 'last_peered': '2024-09-15T21:04:22.480279+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:22.480279+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2815, 'log_dups_size': 0, 'ondisk_log_size': 2815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23050886, 'num_objects': 2814, 'num_object_clones': 0, 'num_object_copies': 11256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2814, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2814, 'num_write_kb': 22511, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2810", 'reported_seq': 2884, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:20.939052+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:20.939052+0000', 'last_peered': '2024-09-15T21:04:20.939052+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:20.939052+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2810, 'log_dups_size': 0, 'ondisk_log_size': 2810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23009926, 'num_objects': 2809, 'num_object_clones': 0, 'num_object_copies': 11236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2809, 'num_write_kb': 22471, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2853", 'reported_seq': 2904, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.511171+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:22.511171+0000', 'last_peered': '2024-09-15T21:04:22.511171+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:22.511171+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2853, 'log_dups_size': 0, 'ondisk_log_size': 2853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23336204, 'num_objects': 2849, 'num_object_clones': 0, 'num_object_copies': 11396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2851, 'num_write_kb': 22804, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:24.959 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2816", 'reported_seq': 2893, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:20.942178+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:20.942178+0000', 'last_peered': '2024-09-15T21:04:20.942178+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:20.942178+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2816, 'log_dups_size': 0, 'ondisk_log_size': 2816, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23041292, 'num_objects': 2813, 'num_object_clones': 0, 'num_object_copies': 11252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2814, 'num_write_kb': 22509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:24.960 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:24.960 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'2882", 'reported_seq': 2945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:22.234117+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:22.234117+0000', 'last_peered': '2024-09-15T21:04:22.234117+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:22.234117+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2882, 'log_dups_size': 0, 'ondisk_log_size': 2882, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23565580, 'num_objects': 2877, 'num_object_clones': 0, 'num_object_copies': 11508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2877, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2880, 'num_write_kb': 23028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 562}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:24.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:25.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 62 20 46973 46953 5.91569 2.71875 0.0700183 0.0263946 2024-09-15T21:04:25.727 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:25.727 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36817/182736 objects degraded (20.148%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":131,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45684,"data_bytes":374204072,"bytes_used":1108709376,"bytes_avail":385438347264,"bytes_total":386547056640,"degraded_objects":36817,"degraded_total":182736,"degraded_ratio":0.20147644689606864,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":121484,"recovering_keys_per_sec":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"write_bytes_sec":3570836,"read_op_per_sec":0,"write_op_per_sec":435},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:25.758 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:25.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:26.396 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:26.396 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:26.431 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-15T21:04:25.923136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1082724,"kb_used_data":578980,"kb_used_omap":14,"kb_used_meta":503665,"kb_avail":376404636,"statfs":{"total":386547056640,"available":385438347264,"internally_reserved":0,"allocated":592875520,"data_stored":592247720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":515753522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,8,20,16,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30695424,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3747,"num_write_kb":29976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002932"},"pg_stats":[{"pgid":"1.f","version":"31'2783","reported_seq":2823,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.524949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:22.524949+0000","last_peered":"2024-09-15T21:04:22.524949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:22.524949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2783,"log_dups_size":0,"ondisk_log_size":2783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22798336,"num_objects":2783,"num_object_clones":0,"num_object_copies":11132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2783,"num_write_kb":22264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'2942","reported_seq":3004,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.384814+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:22.384814+0000","last_peered":"2024-09-15T21:04:22.384814+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:22.384814+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2942,"log_dups_size":0,"ondisk_log_size":2942,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24065292,"num_objects":2938,"num_object_clones":0,"num_object_copies":11752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2940,"num_write_kb":23516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2752","reported_seq":2824,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.941761+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:20.941761+0000","last_peered":"2024-09-15T21:04:20.941761+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:20.941761+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2752,"log_dups_size":0,"ondisk_log_size":2752,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22518406,"num_objects":2749,"num_object_clones":0,"num_object_copies":10996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2751,"num_write_kb":21998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":491}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'2887","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.332801+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:22.332801+0000","last_peered":"2024-09-15T21:04:22.332801+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:22.332801+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2887,"log_dups_size":0,"ondisk_log_size":2887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23614732,"num_objects":2883,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2885,"num_write_kb":23069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":549}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'2900","reported_seq":2950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.515394+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:22.515394+0000","last_peered":"2024-09-15T21:04:22.515394+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:22.515394+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2900,"log_dups_size":0,"ondisk_log_size":2900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23729420,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2898,"num_write_kb":23181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2865","reported_seq":2902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.543672+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:22.543672+0000","last_peered":"2024-09-15T21:04:22.543672+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:22.543672+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2865,"log_dups_size":0,"ondisk_log_size":2865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23470080,"num_objects":2865,"num_object_clones":0,"num_object_copies":11460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2865,"num_write_kb":22920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'2885","reported_seq":2947,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.351421+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:22.351421+0000","last_peered":"2024-09-15T21:04:22.351421+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:22.351421+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2885,"log_dups_size":0,"ondisk_log_size":2885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2883,"num_write_kb":23044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'2989","reported_seq":3052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.289033+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:22.289033+0000","last_peered":"2024-09-15T21:04:22.289033+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:22.289033+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2989,"log_dups_size":0,"ondisk_log_size":2989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24442124,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":23884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2890","reported_seq":2943,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.540684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:22.540684+0000","last_peered":"2024-09-15T21:04:22.540684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:22.540684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2890,"log_dups_size":0,"ondisk_log_size":2890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23603736,"num_objects":2882,"num_object_clones":0,"num_object_copies":11528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2887,"num_write_kb":23073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":550}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2876","reported_seq":2951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.960661+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:20.960661+0000","last_peered":"2024-09-15T21:04:20.960661+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:20.960661+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2876,"log_dups_size":0,"ondisk_log_size":2876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23541004,"num_objects":2874,"num_object_clones":0,"num_object_copies":11496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":22990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":539}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2793","reported_seq":3380,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.497447+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:22.497447+0000","last_peered":"2024-09-15T21:04:22.497447+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:22.497447+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2793,"log_dups_size":0,"ondisk_log_size":2793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22835090,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":22308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":794}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2815","reported_seq":2849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.480279+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:22.480279+0000","last_peered":"2024-09-15T21:04:22.480279+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:22.480279+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2815,"log_dups_size":0,"ondisk_log_size":2815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23050886,"num_objects":2814,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":558}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2810","reported_seq":2884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.939052+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:20.939052+0000","last_peered":"2024-09-15T21:04:20.939052+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:20.939052+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2810,"log_dups_size":0,"ondisk_log_size":2810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23009926,"num_objects":2809,"num_object_clones":0,"num_object_copies":11236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2809,"num_write_kb":22471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2853","reported_seq":2904,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.511171+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:22.511171+0000","last_peered":"2024-09-15T21:04:22.511171+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:22.511171+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23336204,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2851,"num_write_kb":22804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":570}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2816","reported_seq":2893,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:20.942178+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:20.942178+0000","last_peered":"2024-09-15T21:04:20.942178+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:20.942178+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2816,"log_dups_size":0,"ondisk_log_size":2816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23041292,"num_objects":2813,"num_object_clones":0,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2814,"num_write_kb":22509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":532}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'2882","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:22.234117+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:22.234117+0000","last_peered":"2024-09-15T21:04:22.234117+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:22.234117+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23565580,"num_objects":2877,"num_object_clones":0,"num_object_copies":11508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":23028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":562}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":374204072,"num_objects":45684,"num_object_clones":0,"num_object_copies":182736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45711,"num_write_kb":365570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":2179072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":595521536,"data_stored":595521536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45738,"ondisk_log_size":45738,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326716,"kb_used_data":182572,"kb_used_omap":4,"kb_used_meta":144123,"kb_avail":94045124,"statfs":{"total":96636764160,"available":96302206976,"internally_reserved":0,"allocated":186953728,"data_stored":186796778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":147582672},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,0,6,9,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":328876,"kb_used_data":181084,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":94042964,"statfs":{"total":96636764160,"available":96299995136,"internally_reserved":0,"allocated":185430016,"data_stored":185273066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,10,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051590,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104772,"kb_used_data":32884,"kb_used_omap":4,"kb_used_meta":71867,"kb_avail":94267068,"statfs":{"total":96636764160,"available":96529477632,"internally_reserved":0,"allocated":33673216,"data_stored":33516266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":73592527},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":322360,"kb_used_data":182440,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94049480,"statfs":{"total":96636764160,"available":96306667520,"internally_reserved":0,"allocated":186818560,"data_stored":186661610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":143257294},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187510784,"data_stored":187510784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34885632,"data_stored":34885632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185352192,"data_stored":185352192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187772928,"data_stored":187772928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:26.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 63 20 47336 47316 5.8668 2.83594 0.0419886 0.0266206 2024-09-15T21:04:27.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 64 20 47725 47705 5.82262 3.03906 0.0476283 0.0268205 2024-09-15T21:04:28.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 65 20 48115 48095 5.77991 3.04688 0.0778556 0.0270127 2024-09-15T21:04:29.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:29.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 66 20 48496 48476 5.73744 2.97656 0.0220046 0.0272115 2024-09-15T21:04:30.038 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:30.038 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:30.068 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-15T21:04:29.924062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389662376,"num_objects":47571,"num_object_clones":0,"num_object_copies":190284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47598,"num_write_kb":380666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47625,"ondisk_log_size":47625,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1186552,"kb_used_data":610616,"kb_used_omap":14,"kb_used_meta":575857,"kb_avail":376300808,"statfs":{"total":386547056640,"available":385332027392,"internally_reserved":0,"allocated":625270784,"data_stored":624642984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":589678130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,12,10,20,11,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30539776,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002871"},"pg_stats":[{"pgid":"1.f","version":"31'2906","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.544986+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:27.544986+0000","last_peered":"2024-09-15T21:04:27.544986+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:27.544986+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":23248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3065","reported_seq":3125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.343171+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:27.343171+0000","last_peered":"2024-09-15T21:04:27.343171+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:27.343171+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3065,"log_dups_size":0,"ondisk_log_size":3065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25072908,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3063,"num_write_kb":24500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2872","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.918711+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:25.918711+0000","last_peered":"2024-09-15T21:04:25.918711+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:25.918711+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23501446,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2871,"num_write_kb":22958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":611}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3015","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423322+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:27.423322+0000","last_peered":"2024-09-15T21:04:27.423322+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:27.423322+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663308,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3006","reported_seq":3056,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.541333+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:27.541333+0000","last_peered":"2024-09-15T21:04:27.541333+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:27.541333+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597772,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2973","reported_seq":3012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.577413+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:27.577413+0000","last_peered":"2024-09-15T21:04:27.577413+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:27.577413+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3020","reported_seq":3084,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.361909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:27.361909+0000","last_peered":"2024-09-15T21:04:27.361909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:27.361909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24687884,"num_objects":3014,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":24124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":648}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3113","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.312494+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:27.312494+0000","last_peered":"2024-09-15T21:04:27.312494+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:27.312494+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25457932,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2996","reported_seq":3050,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.523894+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:27.523894+0000","last_peered":"2024-09-15T21:04:27.523894+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:27.523894+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24472088,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2993,"num_write_kb":23921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2986","reported_seq":3061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.973049+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:25.973049+0000","last_peered":"2024-09-15T21:04:25.973049+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:25.973049+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2986,"log_dups_size":0,"ondisk_log_size":2986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24442124,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2984,"num_write_kb":23870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2896","reported_seq":3599,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.603107+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:27.603107+0000","last_peered":"2024-09-15T21:04:27.603107+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:27.603107+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23678866,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2918","reported_seq":2952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.414847+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:27.414847+0000","last_peered":"2024-09-15T21:04:27.414847+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:27.414847+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2918,"log_dups_size":0,"ondisk_log_size":2918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23894662,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2917,"num_write_kb":23335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2924","reported_seq":2998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.973413+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:25.973413+0000","last_peered":"2024-09-15T21:04:25.973413+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:25.973413+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23943814,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2982","reported_seq":3031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.587609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:27.587609+0000","last_peered":"2024-09-15T21:04:27.587609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:27.587609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2947","reported_seq":3024,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:26.012188+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:26.012188+0000","last_peered":"2024-09-15T21:04:26.012188+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:26.012188+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2947,"log_dups_size":0,"ondisk_log_size":2947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114444,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3006","reported_seq":3069,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423397+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:27.423397+0000","last_peered":"2024-09-15T21:04:27.423397+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:27.423397+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24581388,"num_objects":3001,"num_object_clones":0,"num_object_copies":12004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389662376,"num_objects":47571,"num_object_clones":0,"num_object_copies":190284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47598,"num_write_kb":380666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626438144,"data_stored":626438144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47625,"ondisk_log_size":47625,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356548,"kb_used_data":190196,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94015292,"statfs":{"total":96636764160,"available":96271659008,"internally_reserved":0,"allocated":194760704,"data_stored":194603754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341840,"kb_used_data":188480,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94030000,"statfs":{"total":96636764160,"available":96286720000,"internally_reserved":0,"allocated":193003520,"data_stored":192846570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136144,"kb_used_data":42048,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94235696,"statfs":{"total":96636764160,"available":96497352704,"internally_reserved":0,"allocated":43057152,"data_stored":42900202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352020,"kb_used_data":189892,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94019820,"statfs":{"total":96636764160,"available":96276295680,"internally_reserved":0,"allocated":194449408,"data_stored":194292458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":165998286},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,4,11,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195174400,"data_stored":195174400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42819584,"data_stored":42819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":193011712,"data_stored":193011712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195432448,"data_stored":195432448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:30.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:30.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 67 20 48838 48818 5.69168 2.67188 0.0633154 0.0274335 2024-09-15T21:04:30.661 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:30.662 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:30.687 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-15T21:04:29.924062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389662376,"num_objects":47571,"num_object_clones":0,"num_object_copies":190284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47598,"num_write_kb":380666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47625,"ondisk_log_size":47625,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1186552,"kb_used_data":610616,"kb_used_omap":14,"kb_used_meta":575857,"kb_avail":376300808,"statfs":{"total":386547056640,"available":385332027392,"internally_reserved":0,"allocated":625270784,"data_stored":624642984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":589678130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,12,10,20,11,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30539776,"num_objects":3728,"num_object_clones":0,"num_object_copies":14912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002871"},"pg_stats":[{"pgid":"1.f","version":"31'2906","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.544986+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:27.544986+0000","last_peered":"2024-09-15T21:04:27.544986+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:27.544986+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":23248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3065","reported_seq":3125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.343171+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:27.343171+0000","last_peered":"2024-09-15T21:04:27.343171+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:27.343171+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3065,"log_dups_size":0,"ondisk_log_size":3065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25072908,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3063,"num_write_kb":24500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2872","reported_seq":2945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.918711+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:25.918711+0000","last_peered":"2024-09-15T21:04:25.918711+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:25.918711+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2872,"log_dups_size":0,"ondisk_log_size":2872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23501446,"num_objects":2869,"num_object_clones":0,"num_object_copies":11476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2871,"num_write_kb":22958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":611}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3015","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423322+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:27.423322+0000","last_peered":"2024-09-15T21:04:27.423322+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:27.423322+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663308,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3006","reported_seq":3056,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.541333+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:27.541333+0000","last_peered":"2024-09-15T21:04:27.541333+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:27.541333+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597772,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2973","reported_seq":3012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.577413+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:27.577413+0000","last_peered":"2024-09-15T21:04:27.577413+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:27.577413+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3020","reported_seq":3084,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.361909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:27.361909+0000","last_peered":"2024-09-15T21:04:27.361909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:27.361909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24687884,"num_objects":3014,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":24124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":648}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3113","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.312494+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:27.312494+0000","last_peered":"2024-09-15T21:04:27.312494+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:27.312494+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25457932,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2996","reported_seq":3050,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.523894+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:27.523894+0000","last_peered":"2024-09-15T21:04:27.523894+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:27.523894+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24472088,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2993,"num_write_kb":23921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'2986","reported_seq":3061,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.973049+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:25.973049+0000","last_peered":"2024-09-15T21:04:25.973049+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:25.973049+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2986,"log_dups_size":0,"ondisk_log_size":2986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24442124,"num_objects":2984,"num_object_clones":0,"num_object_copies":11936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2984,"num_write_kb":23870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":649}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2896","reported_seq":3599,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.603107+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:27.603107+0000","last_peered":"2024-09-15T21:04:27.603107+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:27.603107+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23678866,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2918","reported_seq":2952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.414847+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:27.414847+0000","last_peered":"2024-09-15T21:04:27.414847+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:27.414847+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2918,"log_dups_size":0,"ondisk_log_size":2918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23894662,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2917,"num_write_kb":23335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'2924","reported_seq":2998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:25.973413+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:25.973413+0000","last_peered":"2024-09-15T21:04:25.973413+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:25.973413+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2924,"log_dups_size":0,"ondisk_log_size":2924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23943814,"num_objects":2923,"num_object_clones":0,"num_object_copies":11692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2923,"num_write_kb":23383,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":637}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2982","reported_seq":3031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.587609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:27.587609+0000","last_peered":"2024-09-15T21:04:27.587609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:27.587609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'2947","reported_seq":3024,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:26.012188+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:26.012188+0000","last_peered":"2024-09-15T21:04:26.012188+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:26.012188+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2947,"log_dups_size":0,"ondisk_log_size":2947,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24114444,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2945,"num_write_kb":23557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":663}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3006","reported_seq":3069,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423397+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:27.423397+0000","last_peered":"2024-09-15T21:04:27.423397+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:27.423397+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24581388,"num_objects":3001,"num_object_clones":0,"num_object_copies":12004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":389662376,"num_objects":47571,"num_object_clones":0,"num_object_copies":190284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47598,"num_write_kb":380666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626438144,"data_stored":626438144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":47625,"ondisk_log_size":47625,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356548,"kb_used_data":190196,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94015292,"statfs":{"total":96636764160,"available":96271659008,"internally_reserved":0,"allocated":194760704,"data_stored":194603754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":341840,"kb_used_data":188480,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":94030000,"statfs":{"total":96636764160,"available":96286720000,"internally_reserved":0,"allocated":193003520,"data_stored":192846570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051591,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":136144,"kb_used_data":42048,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94235696,"statfs":{"total":96636764160,"available":96497352704,"internally_reserved":0,"allocated":43057152,"data_stored":42900202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352020,"kb_used_data":189892,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94019820,"statfs":{"total":96636764160,"available":96276295680,"internally_reserved":0,"allocated":194449408,"data_stored":194292458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":165998286},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,4,11,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195174400,"data_stored":195174400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":42819584,"data_stored":42819584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":193011712,"data_stored":193011712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195432448,"data_stored":195432448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'2906", 'reported_seq': 2946, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.544986+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:27.544986+0000', 'last_peered': '2024-09-15T21:04:27.544986+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:27.544986+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2906, 'log_dups_size': 0, 'ondisk_log_size': 2906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23805952, 'num_objects': 2906, 'num_object_clones': 0, 'num_object_copies': 11624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2906, 'num_write_kb': 23248, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 630}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3065", 'reported_seq': 3125, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.343171+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:27.343171+0000', 'last_peered': '2024-09-15T21:04:27.343171+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:27.343171+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3065, 'log_dups_size': 0, 'ondisk_log_size': 3065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25072908, 'num_objects': 3061, 'num_object_clones': 0, 'num_object_copies': 12244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3063, 'num_write_kb': 24500, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 681}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2872", 'reported_seq': 2945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:25.918711+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:25.918711+0000', 'last_peered': '2024-09-15T21:04:25.918711+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:25.918711+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2872, 'log_dups_size': 0, 'ondisk_log_size': 2872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23501446, 'num_objects': 2869, 'num_object_clones': 0, 'num_object_copies': 11476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2871, 'num_write_kb': 22958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 611}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3015", 'reported_seq': 3078, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.423322+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:27.423322+0000', 'last_peered': '2024-09-15T21:04:27.423322+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:27.423322+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3015, 'log_dups_size': 0, 'ondisk_log_size': 3015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24663308, 'num_objects': 3011, 'num_object_clones': 0, 'num_object_copies': 12044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3013, 'num_write_kb': 24093, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:30.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3006", 'reported_seq': 3056, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.541333+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:27.541333+0000', 'last_peered': '2024-09-15T21:04:27.541333+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:27.541333+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3006, 'log_dups_size': 0, 'ondisk_log_size': 3006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24597772, 'num_objects': 3003, 'num_object_clones': 0, 'num_object_copies': 12012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3003, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3004, 'num_write_kb': 24029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'2973", 'reported_seq': 3012, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.577413+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:27.577413+0000', 'last_peered': '2024-09-15T21:04:27.577413+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:27.577413+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2973, 'log_dups_size': 0, 'ondisk_log_size': 2973, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24354816, 'num_objects': 2973, 'num_object_clones': 0, 'num_object_copies': 11892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2973, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2973, 'num_write_kb': 23784, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 657}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3020", 'reported_seq': 3084, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.361909+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:27.361909+0000', 'last_peered': '2024-09-15T21:04:27.361909+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:27.361909+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3020, 'log_dups_size': 0, 'ondisk_log_size': 3020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24687884, 'num_objects': 3014, 'num_object_clones': 0, 'num_object_copies': 12056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3018, 'num_write_kb': 24124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 648}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3113", 'reported_seq': 3176, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.312494+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:27.312494+0000', 'last_peered': '2024-09-15T21:04:27.312494+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:27.312494+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3113, 'log_dups_size': 0, 'ondisk_log_size': 3113, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25457932, 'num_objects': 3108, 'num_object_clones': 0, 'num_object_copies': 12432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3111, 'num_write_kb': 24876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 634}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'2996", 'reported_seq': 3050, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.523894+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:27.523894+0000', 'last_peered': '2024-09-15T21:04:27.523894+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:27.523894+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2996, 'log_dups_size': 0, 'ondisk_log_size': 2996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24472088, 'num_objects': 2988, 'num_object_clones': 0, 'num_object_copies': 11952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2993, 'num_write_kb': 23921, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 656}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:30.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'2986", 'reported_seq': 3061, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:25.973049+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:25.973049+0000', 'last_peered': '2024-09-15T21:04:25.973049+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:25.973049+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2986, 'log_dups_size': 0, 'ondisk_log_size': 2986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24442124, 'num_objects': 2984, 'num_object_clones': 0, 'num_object_copies': 11936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2984, 'num_write_kb': 23870, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 649}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'2896", 'reported_seq': 3599, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.603107+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:27.603107+0000', 'last_peered': '2024-09-15T21:04:27.603107+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:27.603107+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2896, 'log_dups_size': 0, 'ondisk_log_size': 2896, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23678866, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1935, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2893, 'num_write_kb': 23132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 328, 'num_bytes_recovered': 2662400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 956}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'2918", 'reported_seq': 2952, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.414847+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:27.414847+0000', 'last_peered': '2024-09-15T21:04:27.414847+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:27.414847+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2918, 'log_dups_size': 0, 'ondisk_log_size': 2918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23894662, 'num_objects': 2917, 'num_object_clones': 0, 'num_object_copies': 11668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2917, 'num_write_kb': 23335, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 661}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'2924", 'reported_seq': 2998, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:25.973413+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:25.973413+0000', 'last_peered': '2024-09-15T21:04:25.973413+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:25.973413+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2924, 'log_dups_size': 0, 'ondisk_log_size': 2924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23943814, 'num_objects': 2923, 'num_object_clones': 0, 'num_object_copies': 11692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2923, 'num_write_kb': 23383, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 637}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'2982", 'reported_seq': 3031, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.587609+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:27.587609+0000', 'last_peered': '2024-09-15T21:04:27.587609+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:27.587609+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2982, 'log_dups_size': 0, 'ondisk_log_size': 2982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24392972, 'num_objects': 2978, 'num_object_clones': 0, 'num_object_copies': 11912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2980, 'num_write_kb': 23836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 699}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'2947", 'reported_seq': 3024, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:26.012188+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:26.012188+0000', 'last_peered': '2024-09-15T21:04:26.012188+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:26.012188+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2947, 'log_dups_size': 0, 'ondisk_log_size': 2947, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24114444, 'num_objects': 2944, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2945, 'num_write_kb': 23557, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 663}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:30.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3006", 'reported_seq': 3069, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:27.423397+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:27.423397+0000', 'last_peered': '2024-09-15T21:04:27.423397+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:27.423397+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3006, 'log_dups_size': 0, 'ondisk_log_size': 3006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24581388, 'num_objects': 3001, 'num_object_clones': 0, 'num_object_copies': 12004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3001, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3004, 'num_write_kb': 24020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 686}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:30.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:31.434 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:31.434 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36758/190284 objects degraded (19.317%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":137,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47571,"data_bytes":389662376,"bytes_used":1215029248,"bytes_avail":385332027392,"bytes_total":386547056640,"degraded_objects":36758,"degraded_total":190284,"degraded_ratio":0.19317441298269955,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":83265,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"write_bytes_sec":2544372,"read_op_per_sec":0,"write_op_per_sec":310},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:31.460 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:31.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:31.634 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 68 20 49183 49163 5.64762 2.69531 0.0444741 0.0276522 2024-09-15T21:04:32.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:32.112 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:32.145 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":91,"stamp":"2024-09-15T21:04:31.924456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":393381544,"num_objects":48025,"num_object_clones":0,"num_object_copies":192100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48052,"num_write_kb":384298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48079,"ondisk_log_size":48079,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1211248,"kb_used_data":625584,"kb_used_omap":14,"kb_used_meta":585585,"kb_avail":376276112,"statfs":{"total":386547056640,"available":385306738688,"internally_reserved":0,"allocated":640598016,"data_stored":639970216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":599639602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,7,11,11,24,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34258944,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4182,"num_write_kb":33456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002734"},"pg_stats":[{"pgid":"1.f","version":"31'2906","reported_seq":2946,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.544986+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:27.544986+0000","last_peered":"2024-09-15T21:04:27.544986+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:27.544986+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":0,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23805952,"num_objects":2906,"num_object_clones":0,"num_object_copies":11624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2906,"num_write_kb":23248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":630}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3065","reported_seq":3125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.343171+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:27.343171+0000","last_peered":"2024-09-15T21:04:27.343171+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:27.343171+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3065,"log_dups_size":0,"ondisk_log_size":3065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25072908,"num_objects":3061,"num_object_clones":0,"num_object_copies":12244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3063,"num_write_kb":24500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":681}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.992147+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:30.992147+0000","last_peered":"2024-09-15T21:04:30.992147+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:30.992147+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320646,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3015","reported_seq":3078,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423322+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:27.423322+0000","last_peered":"2024-09-15T21:04:27.423322+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:27.423322+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24663308,"num_objects":3011,"num_object_clones":0,"num_object_copies":12044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3013,"num_write_kb":24093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3006","reported_seq":3056,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.541333+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:27.541333+0000","last_peered":"2024-09-15T21:04:27.541333+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:27.541333+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24597772,"num_objects":3003,"num_object_clones":0,"num_object_copies":12012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'2973","reported_seq":3012,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.577413+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:27.577413+0000","last_peered":"2024-09-15T21:04:27.577413+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:27.577413+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2973,"log_dups_size":0,"ondisk_log_size":2973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24354816,"num_objects":2973,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2973,"num_write_kb":23784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3020","reported_seq":3084,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.361909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:27.361909+0000","last_peered":"2024-09-15T21:04:27.361909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:27.361909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24687884,"num_objects":3014,"num_object_clones":0,"num_object_copies":12056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3018,"num_write_kb":24124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":648}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3113","reported_seq":3176,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.312494+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:27.312494+0000","last_peered":"2024-09-15T21:04:27.312494+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:27.312494+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25457932,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3111,"num_write_kb":24876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":634}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'2996","reported_seq":3050,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.523894+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:27.523894+0000","last_peered":"2024-09-15T21:04:27.523894+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:27.523894+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2996,"log_dups_size":0,"ondisk_log_size":2996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24472088,"num_objects":2988,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2993,"num_write_kb":23921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3107","reported_seq":3182,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.995304+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:30.995304+0000","last_peered":"2024-09-15T21:04:30.995304+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:30.995304+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25433356,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'2896","reported_seq":3599,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.603107+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:27.603107+0000","last_peered":"2024-09-15T21:04:27.603107+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:27.603107+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2896,"log_dups_size":0,"ondisk_log_size":2896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23678866,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2893,"num_write_kb":23132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":956}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'2918","reported_seq":2952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.414847+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:27.414847+0000","last_peered":"2024-09-15T21:04:27.414847+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:27.414847+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2918,"log_dups_size":0,"ondisk_log_size":2918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23894662,"num_objects":2917,"num_object_clones":0,"num_object_copies":11668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2917,"num_write_kb":23335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":661}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3042","reported_seq":3116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.980455+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:30.980455+0000","last_peered":"2024-09-15T21:04:30.980455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:30.980455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24910470,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'2982","reported_seq":3031,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.587609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:27.587609+0000","last_peered":"2024-09-15T21:04:27.587609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:27.587609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2982,"log_dups_size":0,"ondisk_log_size":2982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24392972,"num_objects":2978,"num_object_clones":0,"num_object_copies":11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2980,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":699}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3062","reported_seq":3136,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.996995+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:30.996995+0000","last_peered":"2024-09-15T21:04:30.996995+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:30.996995+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3062,"log_dups_size":0,"ondisk_log_size":3062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056524,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3060,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3006","reported_seq":3069,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:27.423397+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:27.423397+0000","last_peered":"2024-09-15T21:04:27.423397+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:27.423397+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3006,"log_dups_size":0,"ondisk_log_size":3006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24581388,"num_objects":3001,"num_object_clones":0,"num_object_copies":12004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3004,"num_write_kb":24020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":686}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":393381544,"num_objects":48025,"num_object_clones":0,"num_object_copies":192100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36758,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48052,"num_write_kb":384298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":352,"num_bytes_recovered":2662400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":641986560,"data_stored":641986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":48079,"ondisk_log_size":48079,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":356548,"kb_used_data":190196,"kb_used_omap":4,"kb_used_meta":166331,"kb_avail":94015292,"statfs":{"total":96636764160,"available":96271659008,"internally_reserved":0,"allocated":194760704,"data_stored":194603754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":170323664},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,1,2,4,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371420,"kb_used_data":195852,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94000420,"statfs":{"total":96636764160,"available":96256430080,"internally_reserved":0,"allocated":200552448,"data_stored":200395498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131260,"kb_used_data":49644,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94240580,"statfs":{"total":96636764160,"available":96502353920,"internally_reserved":0,"allocated":50835456,"data_stored":50678506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":352020,"kb_used_data":189892,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94019820,"statfs":{"total":96636764160,"available":96276295680,"internally_reserved":0,"allocated":194449408,"data_stored":194292458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":165998286},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,3,4,11,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":195174400,"data_stored":195174400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50704384,"data_stored":50704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200675328,"data_stored":200675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":195432448,"data_stored":195432448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:32.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 69 20 49541 49521 5.6063 2.79688 0.0503867 0.0278533 2024-09-15T21:04:33.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 70 20 49925 49905 5.56907 3 0.0485983 0.0280398 2024-09-15T21:04:34.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 71 20 50308 50288 5.53277 2.99219 0.0615312 0.0282265 2024-09-15T21:04:35.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:35.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 72 20 50665 50645 5.49466 2.78906 0.0733625 0.0284194 2024-09-15T21:04:35.751 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:35.752 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:35.778 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-15T21:04:33.925087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1257784,"kb_used_data":640184,"kb_used_omap":14,"kb_used_meta":617521,"kb_avail":376229576,"statfs":{"total":386547056640,"available":385259085824,"internally_reserved":0,"allocated":655548416,"data_stored":654920616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":632342066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,8,16,16,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41558016,"num_objects":5073,"num_object_clones":0,"num_object_copies":20292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5073,"num_write_kb":40584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":179,"num_bytes_recovered":1466368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002920"},"pg_stats":[{"pgid":"1.f","version":"31'3015","reported_seq":3055,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.553327+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:32.553327+0000","last_peered":"2024-09-15T21:04:32.553327+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:32.553327+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3172","reported_seq":3234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.387805+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:32.387805+0000","last_peered":"2024-09-15T21:04:32.387805+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:32.387805+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25949452,"num_objects":3168,"num_object_clones":0,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":25356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.992147+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:30.992147+0000","last_peered":"2024-09-15T21:04:30.992147+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:30.992147+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320646,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3132","reported_seq":3195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.416367+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:32.416367+0000","last_peered":"2024-09-15T21:04:32.416367+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:32.416367+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25621772,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3130,"num_write_kb":25029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3106","reported_seq":3156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.582963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:32.582963+0000","last_peered":"2024-09-15T21:04:32.582963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:32.582963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3106,"log_dups_size":0,"ondisk_log_size":3106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3081","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.628330+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:32.628330+0000","last_peered":"2024-09-15T21:04:32.628330+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:32.628330+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3144","reported_seq":3208,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.424408+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:32.424408+0000","last_peered":"2024-09-15T21:04:32.424408+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:32.424408+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":25116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":772}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3235","reported_seq":3298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.356338+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:32.356338+0000","last_peered":"2024-09-15T21:04:32.356338+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:32.356338+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457356,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":25852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3111","reported_seq":3165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.607721+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:32.607721+0000","last_peered":"2024-09-15T21:04:32.607721+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:32.607721+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25414168,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3107","reported_seq":3182,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.995304+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:30.995304+0000","last_peered":"2024-09-15T21:04:30.995304+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:30.995304+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25433356,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3012","reported_seq":3831,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.609430+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:32.609430+0000","last_peered":"2024-09-15T21:04:32.609430+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:32.609430+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24629138,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3009,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3030","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.640348+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:32.640348+0000","last_peered":"2024-09-15T21:04:32.640348+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:32.640348+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24812166,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3042","reported_seq":3116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.980455+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:30.980455+0000","last_peered":"2024-09-15T21:04:30.980455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:30.980455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24910470,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3095","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.638015+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:32.638015+0000","last_peered":"2024-09-15T21:04:32.638015+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:32.638015+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25318668,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":812}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3062","reported_seq":3136,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.996995+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:30.996995+0000","last_peered":"2024-09-15T21:04:30.996995+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:30.996995+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3062,"log_dups_size":0,"ondisk_log_size":3062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056524,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3060,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3140","reported_seq":3203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.405672+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:32.405672+0000","last_peered":"2024-09-15T21:04:32.405672+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:32.405672+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":656990208,"data_stored":656990208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373704,"kb_used_data":197624,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93998136,"statfs":{"total":96636764160,"available":96254091264,"internally_reserved":0,"allocated":202366976,"data_stored":202210026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,8,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371420,"kb_used_data":195852,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94000420,"statfs":{"total":96636764160,"available":96256430080,"internally_reserved":0,"allocated":200552448,"data_stored":200395498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131260,"kb_used_data":49644,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94240580,"statfs":{"total":96636764160,"available":96502353920,"internally_reserved":0,"allocated":50835456,"data_stored":50678506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381400,"kb_used_data":197064,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93990440,"statfs":{"total":96636764160,"available":96246210560,"internally_reserved":0,"allocated":201793536,"data_stored":201636586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202637312,"data_stored":202637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50704384,"data_stored":50704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200675328,"data_stored":200675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202973184,"data_stored":202973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:35.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:36.367 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:36.368 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:36.396 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-15T21:04:35.925723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1257784,"kb_used_data":640184,"kb_used_omap":14,"kb_used_meta":617521,"kb_avail":376229576,"statfs":{"total":386547056640,"available":385259085824,"internally_reserved":0,"allocated":655548416,"data_stored":654920616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":632342066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,8,16,16,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30457856,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"1.f","version":"31'3015","reported_seq":3055,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.553327+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:32.553327+0000","last_peered":"2024-09-15T21:04:32.553327+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:32.553327+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3172","reported_seq":3234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.387805+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:32.387805+0000","last_peered":"2024-09-15T21:04:32.387805+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:32.387805+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25949452,"num_objects":3168,"num_object_clones":0,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":25356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.992147+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:30.992147+0000","last_peered":"2024-09-15T21:04:30.992147+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:30.992147+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320646,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3132","reported_seq":3195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.416367+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:32.416367+0000","last_peered":"2024-09-15T21:04:32.416367+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:32.416367+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25621772,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3130,"num_write_kb":25029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3106","reported_seq":3156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.582963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:32.582963+0000","last_peered":"2024-09-15T21:04:32.582963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:32.582963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3106,"log_dups_size":0,"ondisk_log_size":3106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3081","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.628330+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:32.628330+0000","last_peered":"2024-09-15T21:04:32.628330+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:32.628330+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3144","reported_seq":3208,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.424408+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:32.424408+0000","last_peered":"2024-09-15T21:04:32.424408+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:32.424408+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":25116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":772}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3235","reported_seq":3298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.356338+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:32.356338+0000","last_peered":"2024-09-15T21:04:32.356338+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:32.356338+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457356,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":25852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3111","reported_seq":3165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.607721+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:32.607721+0000","last_peered":"2024-09-15T21:04:32.607721+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:32.607721+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25414168,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3107","reported_seq":3182,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.995304+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:30.995304+0000","last_peered":"2024-09-15T21:04:30.995304+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:30.995304+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25433356,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3012","reported_seq":3831,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.609430+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:32.609430+0000","last_peered":"2024-09-15T21:04:32.609430+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:32.609430+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24629138,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3009,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3030","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.640348+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:32.640348+0000","last_peered":"2024-09-15T21:04:32.640348+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:32.640348+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24812166,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3042","reported_seq":3116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.980455+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:30.980455+0000","last_peered":"2024-09-15T21:04:30.980455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:30.980455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24910470,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3095","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.638015+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:32.638015+0000","last_peered":"2024-09-15T21:04:32.638015+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:32.638015+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25318668,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":812}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3062","reported_seq":3136,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.996995+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:30.996995+0000","last_peered":"2024-09-15T21:04:30.996995+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:30.996995+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3062,"log_dups_size":0,"ondisk_log_size":3062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056524,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3060,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3140","reported_seq":3203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.405672+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:32.405672+0000","last_peered":"2024-09-15T21:04:32.405672+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:32.405672+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":656990208,"data_stored":656990208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373704,"kb_used_data":197624,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93998136,"statfs":{"total":96636764160,"available":96254091264,"internally_reserved":0,"allocated":202366976,"data_stored":202210026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,8,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371420,"kb_used_data":195852,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94000420,"statfs":{"total":96636764160,"available":96256430080,"internally_reserved":0,"allocated":200552448,"data_stored":200395498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131260,"kb_used_data":49644,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94240580,"statfs":{"total":96636764160,"available":96502353920,"internally_reserved":0,"allocated":50835456,"data_stored":50678506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381400,"kb_used_data":197064,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93990440,"statfs":{"total":96636764160,"available":96246210560,"internally_reserved":0,"allocated":201793536,"data_stored":201636586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202637312,"data_stored":202637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50704384,"data_stored":50704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200675328,"data_stored":200675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202973184,"data_stored":202973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:36.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:36.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3015", 'reported_seq': 3055, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.553327+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:32.553327+0000', 'last_peered': '2024-09-15T21:04:32.553327+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:32.553327+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3015, 'log_dups_size': 0, 'ondisk_log_size': 3015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24698880, 'num_objects': 3015, 'num_object_clones': 0, 'num_object_copies': 12060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3015, 'num_write_kb': 24120, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 739}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3172", 'reported_seq': 3234, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.387805+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:32.387805+0000', 'last_peered': '2024-09-15T21:04:32.387805+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:32.387805+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3172, 'log_dups_size': 0, 'ondisk_log_size': 3172, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25949452, 'num_objects': 3168, 'num_object_clones': 0, 'num_object_copies': 12672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3170, 'num_write_kb': 25356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 788}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'2972", 'reported_seq': 3045, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:30.992147+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:30.992147+0000', 'last_peered': '2024-09-15T21:04:30.992147+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:30.992147+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 2972, 'log_dups_size': 0, 'ondisk_log_size': 2972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24320646, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2971, 'num_write_kb': 23758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 711}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3132", 'reported_seq': 3195, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.416367+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:32.416367+0000', 'last_peered': '2024-09-15T21:04:32.416367+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:32.416367+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3132, 'log_dups_size': 0, 'ondisk_log_size': 3132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25621772, 'num_objects': 3128, 'num_object_clones': 0, 'num_object_copies': 12512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3130, 'num_write_kb': 25029, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3106", 'reported_seq': 3156, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.582963+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:32.582963+0000', 'last_peered': '2024-09-15T21:04:32.582963+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:32.582963+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3106, 'log_dups_size': 0, 'ondisk_log_size': 3106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25416972, 'num_objects': 3103, 'num_object_clones': 0, 'num_object_copies': 12412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3104, 'num_write_kb': 24829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 792}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3081", 'reported_seq': 3117, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.628330+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:32.628330+0000', 'last_peered': '2024-09-15T21:04:32.628330+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:32.628330+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3081, 'log_dups_size': 0, 'ondisk_log_size': 3081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25239552, 'num_objects': 3081, 'num_object_clones': 0, 'num_object_copies': 12324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3081, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3081, 'num_write_kb': 24648, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 765}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3144", 'reported_seq': 3208, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.424408+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:32.424408+0000', 'last_peered': '2024-09-15T21:04:32.424408+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:32.424408+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3144, 'log_dups_size': 0, 'ondisk_log_size': 3144, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25703692, 'num_objects': 3138, 'num_object_clones': 0, 'num_object_copies': 12552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3142, 'num_write_kb': 25116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 772}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3235", 'reported_seq': 3298, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.356338+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:32.356338+0000', 'last_peered': '2024-09-15T21:04:32.356338+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:32.356338+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3235, 'log_dups_size': 0, 'ondisk_log_size': 3235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26457356, 'num_objects': 3230, 'num_object_clones': 0, 'num_object_copies': 12920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3233, 'num_write_kb': 25852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3111", 'reported_seq': 3165, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.607721+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:32.607721+0000', 'last_peered': '2024-09-15T21:04:32.607721+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:32.607721+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3111, 'log_dups_size': 0, 'ondisk_log_size': 3111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25414168, 'num_objects': 3103, 'num_object_clones': 0, 'num_object_copies': 12412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3108, 'num_write_kb': 24841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 771}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3107", 'reported_seq': 3182, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:30.995304+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:30.995304+0000', 'last_peered': '2024-09-15T21:04:30.995304+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:30.995304+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3107, 'log_dups_size': 0, 'ondisk_log_size': 3107, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25433356, 'num_objects': 3105, 'num_object_clones': 0, 'num_object_copies': 12420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3105, 'num_write_kb': 24838, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 770}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3012", 'reported_seq': 3831, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.609430+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:32.609430+0000', 'last_peered': '2024-09-15T21:04:32.609430+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:32.609430+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3012, 'log_dups_size': 0, 'ondisk_log_size': 3012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24629138, 'num_objects': 3007, 'num_object_clones': 0, 'num_object_copies': 12028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1878, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3009, 'num_write_kb': 24060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 385, 'num_bytes_recovered': 3129344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1129}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3030", 'reported_seq': 3064, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.640348+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:32.640348+0000', 'last_peered': '2024-09-15T21:04:32.640348+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:32.640348+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3030, 'log_dups_size': 0, 'ondisk_log_size': 3030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24812166, 'num_objects': 3029, 'num_object_clones': 0, 'num_object_copies': 12116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3029, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3029, 'num_write_kb': 24231, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3042", 'reported_seq': 3116, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:30.980455+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:30.980455+0000', 'last_peered': '2024-09-15T21:04:30.980455+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:30.980455+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3042, 'log_dups_size': 0, 'ondisk_log_size': 3042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24910470, 'num_objects': 3041, 'num_object_clones': 0, 'num_object_copies': 12164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3041, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3041, 'num_write_kb': 24327, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 755}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3095", 'reported_seq': 3145, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.638015+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:32.638015+0000', 'last_peered': '2024-09-15T21:04:32.638015+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:32.638015+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3095, 'log_dups_size': 0, 'ondisk_log_size': 3095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25318668, 'num_objects': 3091, 'num_object_clones': 0, 'num_object_copies': 12364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3093, 'num_write_kb': 24740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3062", 'reported_seq': 3136, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:30.996995+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:30.996995+0000', 'last_peered': '2024-09-15T21:04:30.996995+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:30.996995+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3062, 'log_dups_size': 0, 'ondisk_log_size': 3062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25056524, 'num_objects': 3059, 'num_object_clones': 0, 'num_object_copies': 12236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3060, 'num_write_kb': 24477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 778}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:36.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3140", 'reported_seq': 3203, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:32.405672+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:32.405672+0000', 'last_peered': '2024-09-15T21:04:32.405672+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:32.405672+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3140, 'log_dups_size': 0, 'ondisk_log_size': 3140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25679116, 'num_objects': 3135, 'num_object_clones': 0, 'num_object_copies': 12540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3138, 'num_write_kb': 25092, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:36.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:36.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 73 20 50985 50965 5.45364 2.5 0.0317492 0.0286307 2024-09-15T21:04:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:37.121 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36701/197608 objects degraded (18.573%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":142,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49402,"data_bytes":404661928,"bytes_used":1287970816,"bytes_avail":385259085824,"bytes_total":386547056640,"degraded_objects":36701,"degraded_total":197608,"degraded_ratio":0.18572628638516658,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79169,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":2537506,"read_op_per_sec":0,"write_op_per_sec":309},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:37.146 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:37.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:37.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 74 20 51313 51293 5.41457 2.5625 0.0649473 0.0288392 2024-09-15T21:04:37.738 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:37.738 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:37.772 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-15T21:04:35.925723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1257784,"kb_used_data":640184,"kb_used_omap":14,"kb_used_meta":617521,"kb_avail":376229576,"statfs":{"total":386547056640,"available":385259085824,"internally_reserved":0,"allocated":655548416,"data_stored":654920616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":632342066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,8,16,16,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30457856,"num_objects":3718,"num_object_clones":0,"num_object_copies":14872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3718,"num_write_kb":29744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003063"},"pg_stats":[{"pgid":"1.f","version":"31'3015","reported_seq":3055,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.553327+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:32.553327+0000","last_peered":"2024-09-15T21:04:32.553327+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:32.553327+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3015,"log_dups_size":0,"ondisk_log_size":3015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24698880,"num_objects":3015,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3015,"num_write_kb":24120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3172","reported_seq":3234,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.387805+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:32.387805+0000","last_peered":"2024-09-15T21:04:32.387805+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:32.387805+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3172,"log_dups_size":0,"ondisk_log_size":3172,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25949452,"num_objects":3168,"num_object_clones":0,"num_object_copies":12672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":25356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'2972","reported_seq":3045,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.992147+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:30.992147+0000","last_peered":"2024-09-15T21:04:30.992147+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:30.992147+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":0,"ondisk_log_size":2972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24320646,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":23758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":711}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3132","reported_seq":3195,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.416367+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:32.416367+0000","last_peered":"2024-09-15T21:04:32.416367+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:32.416367+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3132,"log_dups_size":0,"ondisk_log_size":3132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25621772,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3130,"num_write_kb":25029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3106","reported_seq":3156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.582963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:32.582963+0000","last_peered":"2024-09-15T21:04:32.582963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:32.582963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3106,"log_dups_size":0,"ondisk_log_size":3106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25416972,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":24829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":792}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3081","reported_seq":3117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.628330+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:32.628330+0000","last_peered":"2024-09-15T21:04:32.628330+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:32.628330+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3081,"log_dups_size":0,"ondisk_log_size":3081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25239552,"num_objects":3081,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3081,"num_write_kb":24648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3144","reported_seq":3208,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.424408+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:32.424408+0000","last_peered":"2024-09-15T21:04:32.424408+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:32.424408+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25703692,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3142,"num_write_kb":25116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":772}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3235","reported_seq":3298,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.356338+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:32.356338+0000","last_peered":"2024-09-15T21:04:32.356338+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:32.356338+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3235,"log_dups_size":0,"ondisk_log_size":3235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26457356,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":25852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":756}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3111","reported_seq":3165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.607721+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:32.607721+0000","last_peered":"2024-09-15T21:04:32.607721+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:32.607721+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3111,"log_dups_size":0,"ondisk_log_size":3111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25414168,"num_objects":3103,"num_object_clones":0,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3108,"num_write_kb":24841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3107","reported_seq":3182,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.995304+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:30.995304+0000","last_peered":"2024-09-15T21:04:30.995304+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:30.995304+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25433356,"num_objects":3105,"num_object_clones":0,"num_object_copies":12420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3105,"num_write_kb":24838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":770}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3012","reported_seq":3831,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.609430+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:32.609430+0000","last_peered":"2024-09-15T21:04:32.609430+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:32.609430+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3012,"log_dups_size":0,"ondisk_log_size":3012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24629138,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3009,"num_write_kb":24060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1129}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3030","reported_seq":3064,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.640348+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:32.640348+0000","last_peered":"2024-09-15T21:04:32.640348+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:32.640348+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3030,"log_dups_size":0,"ondisk_log_size":3030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24812166,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3029,"num_write_kb":24231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3042","reported_seq":3116,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.980455+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:30.980455+0000","last_peered":"2024-09-15T21:04:30.980455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:30.980455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3042,"log_dups_size":0,"ondisk_log_size":3042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24910470,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":755}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3095","reported_seq":3145,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.638015+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:32.638015+0000","last_peered":"2024-09-15T21:04:32.638015+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:32.638015+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25318668,"num_objects":3091,"num_object_clones":0,"num_object_copies":12364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3093,"num_write_kb":24740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":812}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3062","reported_seq":3136,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:30.996995+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:30.996995+0000","last_peered":"2024-09-15T21:04:30.996995+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:30.996995+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3062,"log_dups_size":0,"ondisk_log_size":3062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25056524,"num_objects":3059,"num_object_clones":0,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3060,"num_write_kb":24477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3140","reported_seq":3203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:32.405672+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:32.405672+0000","last_peered":"2024-09-15T21:04:32.405672+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:32.405672+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3140,"log_dups_size":0,"ondisk_log_size":3140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":404661928,"num_objects":49402,"num_object_clones":0,"num_object_copies":197608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36701,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49429,"num_write_kb":395314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":3129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":656990208,"data_stored":656990208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49456,"ondisk_log_size":49456,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":373704,"kb_used_data":197624,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93998136,"statfs":{"total":96636764160,"available":96254091264,"internally_reserved":0,"allocated":202366976,"data_stored":202210026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,8,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":371420,"kb_used_data":195852,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":94000420,"statfs":{"total":96636764160,"available":96256430080,"internally_reserved":0,"allocated":200552448,"data_stored":200395498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051592,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":131260,"kb_used_data":49644,"kb_used_omap":4,"kb_used_meta":81595,"kb_avail":94240580,"statfs":{"total":96636764160,"available":96502353920,"internally_reserved":0,"allocated":50835456,"data_stored":50678506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":83553999},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,9],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":381400,"kb_used_data":197064,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93990440,"statfs":{"total":96636764160,"available":96246210560,"internally_reserved":0,"allocated":201793536,"data_stored":201636586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202637312,"data_stored":202637312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":50704384,"data_stored":50704384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200675328,"data_stored":200675328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":202973184,"data_stored":202973184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:38.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 75 20 51682 51662 5.38081 2.88281 0.0721769 0.0290241 2024-09-15T21:04:39.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 76 20 52060 52040 5.34886 2.95312 0.0890722 0.0291989 2024-09-15T21:04:40.636 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 77 20 52439 52419 5.31784 2.96094 0.0424054 0.029368 2024-09-15T21:04:40.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:41.398 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:41.398 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:41.428 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-15T21:04:39.926989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":419301032,"num_objects":51189,"num_object_clones":0,"num_object_copies":204756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51216,"num_write_kb":409610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51243,"ondisk_log_size":51243,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1291052,"kb_used_data":669292,"kb_used_omap":14,"kb_used_meta":621681,"kb_avail":376196308,"statfs":{"total":386547056640,"available":385225019392,"internally_reserved":0,"allocated":685355008,"data_stored":684727208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":636601906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,9,12,22,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29638656,"num_objects":3618,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3618,"num_write_kb":28944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003420"},"pg_stats":[{"pgid":"1.f","version":"31'3119","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.620538+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:37.620538+0000","last_peered":"2024-09-15T21:04:37.620538+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:37.620538+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25550848,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":24952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3262","reported_seq":3324,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.388297+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:37.388297+0000","last_peered":"2024-09-15T21:04:37.388297+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:37.388297+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3078","reported_seq":3151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:36.007075+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:36.007075+0000","last_peered":"2024-09-15T21:04:36.007075+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:36.007075+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3078,"log_dups_size":0,"ondisk_log_size":3078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25188998,"num_objects":3075,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3077,"num_write_kb":24606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3244","reported_seq":3307,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.359353+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:37.359353+0000","last_peered":"2024-09-15T21:04:37.359353+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:37.359353+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3242,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3217","reported_seq":3267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.619802+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:37.619802+0000","last_peered":"2024-09-15T21:04:37.619802+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:37.619802+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326284,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3192","reported_seq":3231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.494391+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:37.494391+0000","last_peered":"2024-09-15T21:04:37.494391+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:37.494391+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3192,"log_dups_size":0,"ondisk_log_size":3192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26148864,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3192,"num_write_kb":25536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.463230+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:37.463230+0000","last_peered":"2024-09-15T21:04:37.463230+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:37.463230+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":892}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.374827+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:37.374827+0000","last_peered":"2024-09-15T21:04:37.374827+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:37.374827+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399436,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3214","reported_seq":3268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.642865+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:37.642865+0000","last_peered":"2024-09-15T21:04:37.642865+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:37.642865+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26257944,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":25665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3222","reported_seq":3297,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:35.918950+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:35.918950+0000","last_peered":"2024-09-15T21:04:35.918950+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:35.918950+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3222,"log_dups_size":0,"ondisk_log_size":3222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26375436,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":25758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":885}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3127","reported_seq":4068,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.605144+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:37.605144+0000","last_peered":"2024-09-15T21:04:37.605144+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:37.605144+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25571218,"num_objects":3122,"num_object_clones":0,"num_object_copies":12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":24980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3139","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.652504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:37.652504+0000","last_peered":"2024-09-15T21:04:37.652504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:37.652504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25705094,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3159","reported_seq":3233,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:35.929017+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:35.929017+0000","last_peered":"2024-09-15T21:04:35.929017+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:35.929017+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3159,"log_dups_size":0,"ondisk_log_size":3159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25868934,"num_objects":3158,"num_object_clones":0,"num_object_copies":12632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3158,"num_write_kb":25263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3206","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.555718+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:37.555718+0000","last_peered":"2024-09-15T21:04:37.555718+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:37.555718+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3206,"log_dups_size":0,"ondisk_log_size":3206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3191","reported_seq":3266,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:36.014290+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:36.014290+0000","last_peered":"2024-09-15T21:04:36.014290+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:36.014290+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26113292,"num_objects":3188,"num_object_clones":0,"num_object_copies":12752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":25509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":907}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3259","reported_seq":3318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.479222+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:37.479222+0000","last_peered":"2024-09-15T21:04:37.479222+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:37.479222+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26653964,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":419301032,"num_objects":51189,"num_object_clones":0,"num_object_copies":204756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51216,"num_write_kb":409610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":686723072,"data_stored":686723072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51243,"ondisk_log_size":51243,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385052,"kb_used_data":204812,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93986788,"statfs":{"total":96636764160,"available":96242470912,"internally_reserved":0,"allocated":209727488,"data_stored":209570538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,10,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":382852,"kb_used_data":203124,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93988988,"statfs":{"total":96636764160,"available":96244723712,"internally_reserved":0,"allocated":207998976,"data_stored":207842026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051593,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":142932,"kb_used_data":57156,"kb_used_omap":4,"kb_used_meta":85755,"kb_avail":94228908,"statfs":{"total":96636764160,"available":96490401792,"internally_reserved":0,"allocated":58527744,"data_stored":58370794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":87813839},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,7,10,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380216,"kb_used_data":204200,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93991624,"statfs":{"total":96636764160,"available":96247422976,"internally_reserved":0,"allocated":209100800,"data_stored":208943850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209977344,"data_stored":209977344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":58413056,"data_stored":58413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":208101376,"data_stored":208101376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210231296,"data_stored":210231296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:41.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:41.635 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 78 20 52766 52746 5.28242 2.55469 0.0700807 0.0295656 2024-09-15T21:04:42.015 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:42.016 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:42.040 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-15T21:04:41.927589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422938280,"num_objects":51633,"num_object_clones":0,"num_object_copies":206532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51660,"num_write_kb":413162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51687,"ondisk_log_size":51687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1313916,"kb_used_data":683836,"kb_used_omap":14,"kb_used_meta":630001,"kb_avail":376173444,"statfs":{"total":386547056640,"available":385201606656,"internally_reserved":0,"allocated":700248064,"data_stored":699620264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":645121586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,9,11,11,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33275904,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003526"},"pg_stats":[{"pgid":"1.f","version":"31'3119","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.620538+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:37.620538+0000","last_peered":"2024-09-15T21:04:37.620538+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:37.620538+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25550848,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":24952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3262","reported_seq":3324,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.388297+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:37.388297+0000","last_peered":"2024-09-15T21:04:37.388297+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:37.388297+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3189","reported_seq":3262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.950202+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:40.950202+0000","last_peered":"2024-09-15T21:04:40.950202+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:40.950202+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3244","reported_seq":3307,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.359353+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:37.359353+0000","last_peered":"2024-09-15T21:04:37.359353+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:37.359353+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3242,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3217","reported_seq":3267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.619802+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:37.619802+0000","last_peered":"2024-09-15T21:04:37.619802+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:37.619802+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326284,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3192","reported_seq":3231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.494391+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:37.494391+0000","last_peered":"2024-09-15T21:04:37.494391+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:37.494391+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3192,"log_dups_size":0,"ondisk_log_size":3192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26148864,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3192,"num_write_kb":25536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.463230+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:37.463230+0000","last_peered":"2024-09-15T21:04:37.463230+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:37.463230+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":892}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.374827+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:37.374827+0000","last_peered":"2024-09-15T21:04:37.374827+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:37.374827+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399436,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3214","reported_seq":3268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.642865+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:37.642865+0000","last_peered":"2024-09-15T21:04:37.642865+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:37.642865+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26257944,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":25665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3350","reported_seq":3425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.954136+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:40.954136+0000","last_peered":"2024-09-15T21:04:40.954136+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:40.954136+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3127","reported_seq":4068,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.605144+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:37.605144+0000","last_peered":"2024-09-15T21:04:37.605144+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:37.605144+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25571218,"num_objects":3122,"num_object_clones":0,"num_object_copies":12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":24980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3139","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.652504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:37.652504+0000","last_peered":"2024-09-15T21:04:37.652504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:37.652504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25705094,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3260","reported_seq":3334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:41.036367+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:41.036367+0000","last_peered":"2024-09-15T21:04:41.036367+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:41.036367+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26696326,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3259,"num_write_kb":26071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3206","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.555718+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:37.555718+0000","last_peered":"2024-09-15T21:04:37.555718+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:37.555718+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3206,"log_dups_size":0,"ondisk_log_size":3206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3295","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.992161+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:40.992161+0000","last_peered":"2024-09-15T21:04:40.992161+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:40.992161+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965260,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3259","reported_seq":3318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.479222+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:37.479222+0000","last_peered":"2024-09-15T21:04:37.479222+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:37.479222+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26653964,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422938280,"num_objects":51633,"num_object_clones":0,"num_object_copies":206532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51660,"num_write_kb":413162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":701562880,"data_stored":701562880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51687,"ondisk_log_size":51687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385052,"kb_used_data":204812,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93986788,"statfs":{"total":96636764160,"available":96242470912,"internally_reserved":0,"allocated":209727488,"data_stored":209570538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,10,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394192,"kb_used_data":210304,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93977648,"statfs":{"total":96636764160,"available":96233111552,"internally_reserved":0,"allocated":215351296,"data_stored":215194346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":154456,"kb_used_data":64520,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94217384,"statfs":{"total":96636764160,"available":96478601216,"internally_reserved":0,"allocated":66068480,"data_stored":65911530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380216,"kb_used_data":204200,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93991624,"statfs":{"total":96636764160,"available":96247422976,"internally_reserved":0,"allocated":209100800,"data_stored":208943850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209977344,"data_stored":209977344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65957888,"data_stored":65957888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215396352,"data_stored":215396352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210231296,"data_stored":210231296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:42.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:42.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3119", 'reported_seq': 3159, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.620538+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:37.620538+0000', 'last_peered': '2024-09-15T21:04:37.620538+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:37.620538+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3119, 'log_dups_size': 0, 'ondisk_log_size': 3119, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25550848, 'num_objects': 3119, 'num_object_clones': 0, 'num_object_copies': 12476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3119, 'num_write_kb': 24952, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 843}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.041 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:42.041 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3262", 'reported_seq': 3324, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.388297+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:37.388297+0000', 'last_peered': '2024-09-15T21:04:37.388297+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:37.388297+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3262, 'log_dups_size': 0, 'ondisk_log_size': 3262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26686732, 'num_objects': 3258, 'num_object_clones': 0, 'num_object_copies': 13032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3260, 'num_write_kb': 26076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 878}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3189", 'reported_seq': 3262, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.950202+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:40.950202+0000', 'last_peered': '2024-09-15T21:04:40.950202+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:40.950202+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3189, 'log_dups_size': 0, 'ondisk_log_size': 3189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26098310, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3188, 'num_write_kb': 25494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 928}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3244", 'reported_seq': 3307, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.359353+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:37.359353+0000', 'last_peered': '2024-09-15T21:04:37.359353+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:37.359353+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3244, 'log_dups_size': 0, 'ondisk_log_size': 3244, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26539276, 'num_objects': 3240, 'num_object_clones': 0, 'num_object_copies': 12960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3242, 'num_write_kb': 25925, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 906}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3217", 'reported_seq': 3267, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.619802+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:37.619802+0000', 'last_peered': '2024-09-15T21:04:37.619802+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:37.619802+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3217, 'log_dups_size': 0, 'ondisk_log_size': 3217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26326284, 'num_objects': 3214, 'num_object_clones': 0, 'num_object_copies': 12856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3215, 'num_write_kb': 25717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 903}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3192", 'reported_seq': 3231, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.494391+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:37.494391+0000', 'last_peered': '2024-09-15T21:04:37.494391+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:37.494391+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3192, 'log_dups_size': 0, 'ondisk_log_size': 3192, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26148864, 'num_objects': 3192, 'num_object_clones': 0, 'num_object_copies': 12768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3192, 'num_write_kb': 25536, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 876}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3264", 'reported_seq': 3328, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.463230+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:37.463230+0000', 'last_peered': '2024-09-15T21:04:37.463230+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:37.463230+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3264, 'log_dups_size': 0, 'ondisk_log_size': 3264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26686732, 'num_objects': 3258, 'num_object_clones': 0, 'num_object_copies': 13032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3262, 'num_write_kb': 26076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 892}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:42.042 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3350", 'reported_seq': 3412, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.374827+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:37.374827+0000', 'last_peered': '2024-09-15T21:04:37.374827+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:37.374827+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27399436, 'num_objects': 3345, 'num_object_clones': 0, 'num_object_copies': 13380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26772, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 871}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3214", 'reported_seq': 3268, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.642865+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:37.642865+0000', 'last_peered': '2024-09-15T21:04:37.642865+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:37.642865+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3214, 'log_dups_size': 0, 'ondisk_log_size': 3214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26257944, 'num_objects': 3206, 'num_object_clones': 0, 'num_object_copies': 12824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3211, 'num_write_kb': 25665, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 874}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3350", 'reported_seq': 3425, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.954136+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:40.954136+0000', 'last_peered': '2024-09-15T21:04:40.954136+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:40.954136+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27424012, 'num_objects': 3348, 'num_object_clones': 0, 'num_object_copies': 13392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1013}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3127", 'reported_seq': 4068, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.605144+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:37.605144+0000', 'last_peered': '2024-09-15T21:04:37.605144+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:37.605144+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3127, 'log_dups_size': 0, 'ondisk_log_size': 3127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25571218, 'num_objects': 3122, 'num_object_clones': 0, 'num_object_copies': 12488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1817, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3124, 'num_write_kb': 24980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 446, 'num_bytes_recovered': 3629056, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1305}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3139", 'reported_seq': 3173, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.652504+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:37.652504+0000', 'last_peered': '2024-09-15T21:04:37.652504+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:37.652504+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3139, 'log_dups_size': 0, 'ondisk_log_size': 3139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25705094, 'num_objects': 3138, 'num_object_clones': 0, 'num_object_copies': 12552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3138, 'num_write_kb': 25103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:42.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3260", 'reported_seq': 3334, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:41.036367+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:41.036367+0000', 'last_peered': '2024-09-15T21:04:41.036367+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:41.036367+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3260, 'log_dups_size': 0, 'ondisk_log_size': 3260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26696326, 'num_objects': 3259, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3259, 'num_write_kb': 26071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3206", 'reported_seq': 3256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.555718+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:37.555718+0000', 'last_peered': '2024-09-15T21:04:37.555718+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:37.555718+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3206, 'log_dups_size': 0, 'ondisk_log_size': 3206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26227980, 'num_objects': 3202, 'num_object_clones': 0, 'num_object_copies': 12808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3204, 'num_write_kb': 25628, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 923}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3295", 'reported_seq': 3372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.992161+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:40.992161+0000', 'last_peered': '2024-09-15T21:04:40.992161+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:40.992161+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3295, 'log_dups_size': 0, 'ondisk_log_size': 3295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26965260, 'num_objects': 3292, 'num_object_clones': 0, 'num_object_copies': 13168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3293, 'num_write_kb': 26341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:42.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3259", 'reported_seq': 3318, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:37.479222+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:37.479222+0000', 'last_peered': '2024-09-15T21:04:37.479222+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:37.479222+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3259, 'log_dups_size': 0, 'ondisk_log_size': 3259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26653964, 'num_objects': 3254, 'num_object_clones': 0, 'num_object_copies': 13016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3257, 'num_write_kb': 26044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:42.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:42.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 79 20 53100 53080 5.24852 2.60938 0.051416 0.0297498 2024-09-15T21:04:42.762 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:42.762 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36640/206532 objects degraded (17.741%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":148,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51633,"data_bytes":422938280,"bytes_used":1345449984,"bytes_avail":385201606656,"bytes_total":386547056640,"degraded_objects":36640,"degraded_total":206532,"degraded_ratio":0.17740592256889973,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":80530,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"write_bytes_sec":2772177,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:42.795 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:42.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:43.387 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:43.387 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:43.424 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-15T21:04:41.927589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422938280,"num_objects":51633,"num_object_clones":0,"num_object_copies":206532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51660,"num_write_kb":413162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51687,"ondisk_log_size":51687,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1313916,"kb_used_data":683836,"kb_used_omap":14,"kb_used_meta":630001,"kb_avail":376173444,"statfs":{"total":386547056640,"available":385201606656,"internally_reserved":0,"allocated":700248064,"data_stored":699620264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":645121586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,0,2,9,11,11,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33275904,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4062,"num_write_kb":32496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003526"},"pg_stats":[{"pgid":"1.f","version":"31'3119","reported_seq":3159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.620538+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:37.620538+0000","last_peered":"2024-09-15T21:04:37.620538+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:37.620538+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25550848,"num_objects":3119,"num_object_clones":0,"num_object_copies":12476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":24952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3262","reported_seq":3324,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.388297+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:37.388297+0000","last_peered":"2024-09-15T21:04:37.388297+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:37.388297+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3262,"log_dups_size":0,"ondisk_log_size":3262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3189","reported_seq":3262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.950202+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:40.950202+0000","last_peered":"2024-09-15T21:04:40.950202+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:40.950202+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3244","reported_seq":3307,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.359353+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:37.359353+0000","last_peered":"2024-09-15T21:04:37.359353+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:37.359353+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3244,"log_dups_size":0,"ondisk_log_size":3244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26539276,"num_objects":3240,"num_object_clones":0,"num_object_copies":12960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3242,"num_write_kb":25925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":906}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3217","reported_seq":3267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.619802+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:37.619802+0000","last_peered":"2024-09-15T21:04:37.619802+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:37.619802+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3217,"log_dups_size":0,"ondisk_log_size":3217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326284,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3192","reported_seq":3231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.494391+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:37.494391+0000","last_peered":"2024-09-15T21:04:37.494391+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:37.494391+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3192,"log_dups_size":0,"ondisk_log_size":3192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26148864,"num_objects":3192,"num_object_clones":0,"num_object_copies":12768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3192,"num_write_kb":25536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":876}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3264","reported_seq":3328,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.463230+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:37.463230+0000","last_peered":"2024-09-15T21:04:37.463230+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:37.463230+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26686732,"num_objects":3258,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3262,"num_write_kb":26076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":892}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3350","reported_seq":3412,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.374827+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:37.374827+0000","last_peered":"2024-09-15T21:04:37.374827+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:37.374827+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27399436,"num_objects":3345,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3214","reported_seq":3268,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.642865+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:37.642865+0000","last_peered":"2024-09-15T21:04:37.642865+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:37.642865+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3214,"log_dups_size":0,"ondisk_log_size":3214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26257944,"num_objects":3206,"num_object_clones":0,"num_object_copies":12824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3211,"num_write_kb":25665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":874}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3350","reported_seq":3425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.954136+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:40.954136+0000","last_peered":"2024-09-15T21:04:40.954136+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:40.954136+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3127","reported_seq":4068,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.605144+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:37.605144+0000","last_peered":"2024-09-15T21:04:37.605144+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:37.605144+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3127,"log_dups_size":0,"ondisk_log_size":3127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25571218,"num_objects":3122,"num_object_clones":0,"num_object_copies":12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3124,"num_write_kb":24980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3139","reported_seq":3173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.652504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:37.652504+0000","last_peered":"2024-09-15T21:04:37.652504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:37.652504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25705094,"num_objects":3138,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3138,"num_write_kb":25103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3260","reported_seq":3334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:41.036367+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:41.036367+0000","last_peered":"2024-09-15T21:04:41.036367+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:41.036367+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26696326,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3259,"num_write_kb":26071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3206","reported_seq":3256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.555718+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:37.555718+0000","last_peered":"2024-09-15T21:04:37.555718+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:37.555718+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3206,"log_dups_size":0,"ondisk_log_size":3206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26227980,"num_objects":3202,"num_object_clones":0,"num_object_copies":12808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3204,"num_write_kb":25628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":923}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3295","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.992161+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:40.992161+0000","last_peered":"2024-09-15T21:04:40.992161+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:40.992161+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965260,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3259","reported_seq":3318,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:37.479222+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:37.479222+0000","last_peered":"2024-09-15T21:04:37.479222+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:37.479222+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26653964,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3257,"num_write_kb":26044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":422938280,"num_objects":51633,"num_object_clones":0,"num_object_copies":206532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51660,"num_write_kb":413162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":470,"num_bytes_recovered":3629056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":701562880,"data_stored":701562880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51687,"ondisk_log_size":51687,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":385052,"kb_used_data":204812,"kb_used_omap":4,"kb_used_meta":180219,"kb_avail":93986788,"statfs":{"total":96636764160,"available":96242470912,"internally_reserved":0,"allocated":209727488,"data_stored":209570538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":184544976},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,1,10,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394192,"kb_used_data":210304,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93977648,"statfs":{"total":96636764160,"available":96233111552,"internally_reserved":0,"allocated":215351296,"data_stored":215194346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":154456,"kb_used_data":64520,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94217384,"statfs":{"total":96636764160,"available":96478601216,"internally_reserved":0,"allocated":66068480,"data_stored":65911530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":380216,"kb_used_data":204200,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93991624,"statfs":{"total":96636764160,"available":96247422976,"internally_reserved":0,"allocated":209100800,"data_stored":208943850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209977344,"data_stored":209977344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65957888,"data_stored":65957888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215396352,"data_stored":215396352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":210231296,"data_stored":210231296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:43.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:04:43.634975+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0299411 2024-09-15T21:04:43.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:04:43.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 80 20 53436 53416 5.21572 2.625 0.0605382 0.0299411 2024-09-15T21:04:44.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 81 20 53810 53790 5.1874 2.92188 0.0600343 0.0301092 2024-09-15T21:04:45.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 82 20 54177 54157 5.1591 2.86719 0.0822035 0.0302717 2024-09-15T21:04:46.426 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:46.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 83 20 54535 54515 5.13064 2.79688 0.0932403 0.0304369 2024-09-15T21:04:47.017 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:47.017 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:47.052 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-15T21:04:45.928932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434054824,"num_objects":52990,"num_object_clones":0,"num_object_copies":211960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53017,"num_write_kb":424018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53044,"ondisk_log_size":53044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1336668,"kb_used_data":698268,"kb_used_omap":14,"kb_used_meta":638321,"kb_avail":376150692,"statfs":{"total":386547056640,"available":385178308608,"internally_reserved":0,"allocated":715026432,"data_stored":714398632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":653641266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,6,4,15,4,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29392896,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3588,"num_write_kb":28704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003845"},"pg_stats":[{"pgid":"1.f","version":"31'3248","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.725128+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:42.725128+0000","last_peered":"2024-09-15T21:04:42.725128+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:42.725128+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3248,"num_write_kb":25984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3358","reported_seq":3419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.472513+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:42.472513+0000","last_peered":"2024-09-15T21:04:42.472513+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:42.472513+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473164,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":26844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":974}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3189","reported_seq":3262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.950202+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:40.950202+0000","last_peered":"2024-09-15T21:04:40.950202+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:40.950202+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3354","reported_seq":3417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.252909+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:42.252909+0000","last_peered":"2024-09-15T21:04:42.252909+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:42.252909+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440396,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":26805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3346","reported_seq":3396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.553828+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:42.553828+0000","last_peered":"2024-09-15T21:04:42.553828+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:42.553828+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27383052,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3344,"num_write_kb":26749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3293","reported_seq":3332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.649759+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:42.649759+0000","last_peered":"2024-09-15T21:04:42.649759+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:42.649759+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26976256,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3367","reported_seq":3431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.058915+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:42.058915+0000","last_peered":"2024-09-15T21:04:42.058915+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:42.058915+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3367,"log_dups_size":0,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530508,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3481","reported_seq":3544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.384284+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:42.384284+0000","last_peered":"2024-09-15T21:04:42.384284+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:42.384284+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28472588,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1002}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3309","reported_seq":3363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.703734+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:42.703734+0000","last_peered":"2024-09-15T21:04:42.703734+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:42.703734+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27036184,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3350","reported_seq":3425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.954136+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:40.954136+0000","last_peered":"2024-09-15T21:04:40.954136+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:40.954136+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3247","reported_seq":4315,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.759583+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:42.759583+0000","last_peered":"2024-09-15T21:04:42.759583+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:42.759583+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554258,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":25940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3243","reported_seq":3277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.750650+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:42.750650+0000","last_peered":"2024-09-15T21:04:42.750650+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:42.750650+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26557062,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3242,"num_write_kb":25935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":986}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3260","reported_seq":3334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:41.036367+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:41.036367+0000","last_peered":"2024-09-15T21:04:41.036367+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:41.036367+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26696326,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3259,"num_write_kb":26071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3322","reported_seq":3373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.622695+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:42.622695+0000","last_peered":"2024-09-15T21:04:42.622695+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:42.622695+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27178252,"num_objects":3318,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3295","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.992161+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:40.992161+0000","last_peered":"2024-09-15T21:04:40.992161+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:40.992161+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965260,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3382","reported_seq":3445,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.364925+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:42.364925+0000","last_peered":"2024-09-15T21:04:42.364925+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:42.364925+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27661580,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":27028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":434054824,"num_objects":52990,"num_object_clones":0,"num_object_copies":211960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53017,"num_write_kb":424018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":716353536,"data_stored":716353536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53044,"ondisk_log_size":53044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396500,"kb_used_data":212100,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93975340,"statfs":{"total":96636764160,"available":96230748160,"internally_reserved":0,"allocated":217190400,"data_stored":217033450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,1,2,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394192,"kb_used_data":210304,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93977648,"statfs":{"total":96636764160,"available":96233111552,"internally_reserved":0,"allocated":215351296,"data_stored":215194346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":154456,"kb_used_data":64520,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94217384,"statfs":{"total":96636764160,"available":96478601216,"internally_reserved":0,"allocated":66068480,"data_stored":65911530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391520,"kb_used_data":211344,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93980320,"statfs":{"total":96636764160,"available":96235847680,"internally_reserved":0,"allocated":216416256,"data_stored":216259306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":184479438},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217329664,"data_stored":217329664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65957888,"data_stored":65957888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215396352,"data_stored":215396352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217669632,"data_stored":217669632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:47.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:47.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 84 20 54857 54837 5.09951 2.51562 0.0793853 0.0306236 2024-09-15T21:04:47.694 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:47.694 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:47.718 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-15T21:04:45.928932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434054824,"num_objects":52990,"num_object_clones":0,"num_object_copies":211960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53017,"num_write_kb":424018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53044,"ondisk_log_size":53044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1336668,"kb_used_data":698268,"kb_used_omap":14,"kb_used_meta":638321,"kb_avail":376150692,"statfs":{"total":386547056640,"available":385178308608,"internally_reserved":0,"allocated":715026432,"data_stored":714398632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":653641266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,3,6,4,15,4,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29392896,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3588,"num_write_kb":28704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003845"},"pg_stats":[{"pgid":"1.f","version":"31'3248","reported_seq":3288,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.725128+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:42.725128+0000","last_peered":"2024-09-15T21:04:42.725128+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:42.725128+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3248,"log_dups_size":0,"ondisk_log_size":3248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26607616,"num_objects":3248,"num_object_clones":0,"num_object_copies":12992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3248,"num_write_kb":25984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":972}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3358","reported_seq":3419,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.472513+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:42.472513+0000","last_peered":"2024-09-15T21:04:42.472513+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:42.472513+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3358,"log_dups_size":0,"ondisk_log_size":3358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27473164,"num_objects":3354,"num_object_clones":0,"num_object_copies":13416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3356,"num_write_kb":26844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":974}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3189","reported_seq":3262,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.950202+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:40.950202+0000","last_peered":"2024-09-15T21:04:40.950202+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:40.950202+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26098310,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3188,"num_write_kb":25494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3354","reported_seq":3417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.252909+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:42.252909+0000","last_peered":"2024-09-15T21:04:42.252909+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:42.252909+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3354,"log_dups_size":0,"ondisk_log_size":3354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27440396,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":26805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1016}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3346","reported_seq":3396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.553828+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:42.553828+0000","last_peered":"2024-09-15T21:04:42.553828+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:42.553828+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3346,"log_dups_size":0,"ondisk_log_size":3346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27383052,"num_objects":3343,"num_object_clones":0,"num_object_copies":13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3344,"num_write_kb":26749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1032}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3293","reported_seq":3332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.649759+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:42.649759+0000","last_peered":"2024-09-15T21:04:42.649759+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:42.649759+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3293,"log_dups_size":0,"ondisk_log_size":3293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26976256,"num_objects":3293,"num_object_clones":0,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3367","reported_seq":3431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.058915+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:42.058915+0000","last_peered":"2024-09-15T21:04:42.058915+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:42.058915+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3367,"log_dups_size":0,"ondisk_log_size":3367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27530508,"num_objects":3361,"num_object_clones":0,"num_object_copies":13444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3365,"num_write_kb":26900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":995}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3481","reported_seq":3544,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.384284+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:42.384284+0000","last_peered":"2024-09-15T21:04:42.384284+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:42.384284+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28472588,"num_objects":3476,"num_object_clones":0,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3479,"num_write_kb":27820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1002}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3309","reported_seq":3363,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.703734+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:42.703734+0000","last_peered":"2024-09-15T21:04:42.703734+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:42.703734+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27036184,"num_objects":3301,"num_object_clones":0,"num_object_copies":13204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3306,"num_write_kb":26425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3350","reported_seq":3425,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.954136+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:40.954136+0000","last_peered":"2024-09-15T21:04:40.954136+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:40.954136+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3348,"num_write_kb":26782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1013}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3247","reported_seq":4315,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.759583+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:42.759583+0000","last_peered":"2024-09-15T21:04:42.759583+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:42.759583+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26554258,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1753,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3244,"num_write_kb":25940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":510,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3243","reported_seq":3277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.750650+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:42.750650+0000","last_peered":"2024-09-15T21:04:42.750650+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:42.750650+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3243,"log_dups_size":0,"ondisk_log_size":3243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26557062,"num_objects":3242,"num_object_clones":0,"num_object_copies":12968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3242,"num_write_kb":25935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":986}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3260","reported_seq":3334,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:41.036367+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:41.036367+0000","last_peered":"2024-09-15T21:04:41.036367+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:41.036367+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26696326,"num_objects":3259,"num_object_clones":0,"num_object_copies":13036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3259,"num_write_kb":26071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3322","reported_seq":3373,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.622695+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:42.622695+0000","last_peered":"2024-09-15T21:04:42.622695+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:42.622695+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3322,"log_dups_size":0,"ondisk_log_size":3322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27178252,"num_objects":3318,"num_object_clones":0,"num_object_copies":13272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3320,"num_write_kb":26556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1039}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3295","reported_seq":3372,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:40.992161+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:40.992161+0000","last_peered":"2024-09-15T21:04:40.992161+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:40.992161+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3295,"log_dups_size":0,"ondisk_log_size":3295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26965260,"num_objects":3292,"num_object_clones":0,"num_object_copies":13168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3293,"num_write_kb":26341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1011}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3382","reported_seq":3445,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:42.364925+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:42.364925+0000","last_peered":"2024-09-15T21:04:42.364925+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:42.364925+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27661580,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3380,"num_write_kb":27028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1062}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":434054824,"num_objects":52990,"num_object_clones":0,"num_object_copies":211960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36576,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53017,"num_write_kb":424018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":4153344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":716353536,"data_stored":716353536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53044,"ondisk_log_size":53044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396500,"kb_used_data":212100,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93975340,"statfs":{"total":96636764160,"available":96230748160,"internally_reserved":0,"allocated":217190400,"data_stored":217033450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,1,2,1,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394192,"kb_used_data":210304,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93977648,"statfs":{"total":96636764160,"available":96233111552,"internally_reserved":0,"allocated":215351296,"data_stored":215194346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,0,3,0,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051594,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":154456,"kb_used_data":64520,"kb_used_omap":4,"kb_used_meta":89915,"kb_avail":94217384,"statfs":{"total":96636764160,"available":96478601216,"internally_reserved":0,"allocated":66068480,"data_stored":65911530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":92073679},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391520,"kb_used_data":211344,"kb_used_omap":4,"kb_used_meta":180155,"kb_avail":93980320,"statfs":{"total":96636764160,"available":96235847680,"internally_reserved":0,"allocated":216416256,"data_stored":216259306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":184479438},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217329664,"data_stored":217329664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65957888,"data_stored":65957888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215396352,"data_stored":215396352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217669632,"data_stored":217669632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:47.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:47.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3248", 'reported_seq': 3288, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.725128+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:42.725128+0000', 'last_peered': '2024-09-15T21:04:42.725128+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:42.725128+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3248, 'log_dups_size': 0, 'ondisk_log_size': 3248, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26607616, 'num_objects': 3248, 'num_object_clones': 0, 'num_object_copies': 12992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3248, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3248, 'num_write_kb': 25984, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 972}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.719 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:47.719 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3358", 'reported_seq': 3419, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.472513+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:42.472513+0000', 'last_peered': '2024-09-15T21:04:42.472513+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:42.472513+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3358, 'log_dups_size': 0, 'ondisk_log_size': 3358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27473164, 'num_objects': 3354, 'num_object_clones': 0, 'num_object_copies': 13416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3356, 'num_write_kb': 26844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 974}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3189", 'reported_seq': 3262, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.950202+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:40.950202+0000', 'last_peered': '2024-09-15T21:04:40.950202+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:40.950202+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3189, 'log_dups_size': 0, 'ondisk_log_size': 3189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26098310, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3188, 'num_write_kb': 25494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 928}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3354", 'reported_seq': 3417, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.252909+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:42.252909+0000', 'last_peered': '2024-09-15T21:04:42.252909+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:42.252909+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3354, 'log_dups_size': 0, 'ondisk_log_size': 3354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27440396, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 13400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3352, 'num_write_kb': 26805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1016}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3346", 'reported_seq': 3396, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.553828+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:42.553828+0000', 'last_peered': '2024-09-15T21:04:42.553828+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:42.553828+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3346, 'log_dups_size': 0, 'ondisk_log_size': 3346, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27383052, 'num_objects': 3343, 'num_object_clones': 0, 'num_object_copies': 13372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3344, 'num_write_kb': 26749, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1032}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3293", 'reported_seq': 3332, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.649759+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:42.649759+0000', 'last_peered': '2024-09-15T21:04:42.649759+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:42.649759+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3293, 'log_dups_size': 0, 'ondisk_log_size': 3293, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26976256, 'num_objects': 3293, 'num_object_clones': 0, 'num_object_copies': 13172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3293, 'num_write_kb': 26344, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 977}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3367", 'reported_seq': 3431, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.058915+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:42.058915+0000', 'last_peered': '2024-09-15T21:04:42.058915+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:42.058915+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3367, 'log_dups_size': 0, 'ondisk_log_size': 3367, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27530508, 'num_objects': 3361, 'num_object_clones': 0, 'num_object_copies': 13444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3361, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3365, 'num_write_kb': 26900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 995}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:47.720 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3481", 'reported_seq': 3544, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.384284+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:42.384284+0000', 'last_peered': '2024-09-15T21:04:42.384284+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:42.384284+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3481, 'log_dups_size': 0, 'ondisk_log_size': 3481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28472588, 'num_objects': 3476, 'num_object_clones': 0, 'num_object_copies': 13904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3476, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3479, 'num_write_kb': 27820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1002}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3309", 'reported_seq': 3363, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.703734+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:42.703734+0000', 'last_peered': '2024-09-15T21:04:42.703734+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:42.703734+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3309, 'log_dups_size': 0, 'ondisk_log_size': 3309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27036184, 'num_objects': 3301, 'num_object_clones': 0, 'num_object_copies': 13204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3306, 'num_write_kb': 26425, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 969}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3350", 'reported_seq': 3425, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.954136+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:40.954136+0000', 'last_peered': '2024-09-15T21:04:40.954136+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:40.954136+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27424012, 'num_objects': 3348, 'num_object_clones': 0, 'num_object_copies': 13392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3348, 'num_write_kb': 26782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1013}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3247", 'reported_seq': 4315, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.759583+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:42.759583+0000', 'last_peered': '2024-09-15T21:04:42.759583+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:42.759583+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3247, 'log_dups_size': 0, 'ondisk_log_size': 3247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26554258, 'num_objects': 3242, 'num_object_clones': 0, 'num_object_copies': 12968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1753, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3244, 'num_write_kb': 25940, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 510, 'num_bytes_recovered': 4153344, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1489}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3243", 'reported_seq': 3277, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.750650+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:42.750650+0000', 'last_peered': '2024-09-15T21:04:42.750650+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:42.750650+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3243, 'log_dups_size': 0, 'ondisk_log_size': 3243, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26557062, 'num_objects': 3242, 'num_object_clones': 0, 'num_object_copies': 12968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3242, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3242, 'num_write_kb': 25935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 986}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:47.721 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3260", 'reported_seq': 3334, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:41.036367+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:41.036367+0000', 'last_peered': '2024-09-15T21:04:41.036367+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:41.036367+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3260, 'log_dups_size': 0, 'ondisk_log_size': 3260, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26696326, 'num_objects': 3259, 'num_object_clones': 0, 'num_object_copies': 13036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3259, 'num_write_kb': 26071, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3322", 'reported_seq': 3373, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.622695+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:42.622695+0000', 'last_peered': '2024-09-15T21:04:42.622695+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:42.622695+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3322, 'log_dups_size': 0, 'ondisk_log_size': 3322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27178252, 'num_objects': 3318, 'num_object_clones': 0, 'num_object_copies': 13272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3318, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3320, 'num_write_kb': 26556, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1039}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3295", 'reported_seq': 3372, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:40.992161+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:40.992161+0000', 'last_peered': '2024-09-15T21:04:40.992161+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:40.992161+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3295, 'log_dups_size': 0, 'ondisk_log_size': 3295, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26965260, 'num_objects': 3292, 'num_object_clones': 0, 'num_object_copies': 13168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3293, 'num_write_kb': 26341, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:47.722 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3382", 'reported_seq': 3445, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:42.364925+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:42.364925+0000', 'last_peered': '2024-09-15T21:04:42.364925+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:42.364925+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3382, 'log_dups_size': 0, 'ondisk_log_size': 3382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27661580, 'num_objects': 3377, 'num_object_clones': 0, 'num_object_copies': 13508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3380, 'num_write_kb': 27028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1062}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:47.723 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:48.511 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:48.511 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36518/219100 objects degraded (16.667%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":154,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54775,"data_bytes":448677544,"bytes_used":1384833024,"bytes_avail":385162223616,"bytes_total":386547056640,"degraded_objects":36518,"degraded_total":219100,"degraded_ratio":0.16667275216795985,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":124889,"recovering_keys_per_sec":0,"num_objects_recovered":183,"num_bytes_recovered":1499136,"num_keys_recovered":0,"write_bytes_sec":3666834,"read_op_per_sec":0,"write_op_per_sec":447},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:48.534 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:48.534 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:48.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 85 20 55150 55130 5.06644 2.28906 0.0509764 0.0308224 2024-09-15T21:04:49.160 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:49.161 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:49.199 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-15T21:04:47.929433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":448677544,"num_objects":54775,"num_object_clones":0,"num_object_copies":219100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54802,"num_write_kb":438298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54829,"ondisk_log_size":54829,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1352376,"kb_used_data":726456,"kb_used_omap":14,"kb_used_meta":625841,"kb_avail":376134984,"statfs":{"total":386547056640,"available":385162223616,"internally_reserved":0,"allocated":743890944,"data_stored":743263144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":640861746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,8,5,12,19,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44015616,"num_objects":5373,"num_object_clones":0,"num_object_copies":21492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5373,"num_write_kb":42984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":1499136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003710"},"pg_stats":[{"pgid":"1.f","version":"31'3350","reported_seq":3390,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.679328+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:47.679328+0000","last_peered":"2024-09-15T21:04:47.679328+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:47.679328+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3463","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.452446+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:47.452446+0000","last_peered":"2024-09-15T21:04:47.452446+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:47.452446+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3463,"log_dups_size":0,"ondisk_log_size":3463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28333324,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":27684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3311","reported_seq":3384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:46.074120+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:46.074120+0000","last_peered":"2024-09-15T21:04:46.074120+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:46.074120+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3311,"log_dups_size":0,"ondisk_log_size":3311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27097734,"num_objects":3308,"num_object_clones":0,"num_object_copies":13232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":26470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1050}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3478","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.414731+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:47.414731+0000","last_peered":"2024-09-15T21:04:47.414731+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:47.414731+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456204,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3460","reported_seq":3510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.775469+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:47.775469+0000","last_peered":"2024-09-15T21:04:47.775469+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:47.775469+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3399","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.697733+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:47.697733+0000","last_peered":"2024-09-15T21:04:47.697733+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:47.697733+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3479","reported_seq":3543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.408330+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:47.408330+0000","last_peered":"2024-09-15T21:04:47.408330+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:47.408330+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28448012,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1107}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3592","reported_seq":3655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.342342+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:47.342342+0000","last_peered":"2024-09-15T21:04:47.342342+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:47.342342+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3422","reported_seq":3476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.634330+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:47.634330+0000","last_peered":"2024-09-15T21:04:47.634330+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:47.634330+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27961880,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":27329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3444","reported_seq":3520,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:46.102717+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:46.102717+0000","last_peered":"2024-09-15T21:04:46.102717+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:46.102717+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3444,"log_dups_size":0,"ondisk_log_size":3444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28194060,"num_objects":3442,"num_object_clones":0,"num_object_copies":13768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":27534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1107}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3374","reported_seq":4559,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.766432+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:47.766432+0000","last_peered":"2024-09-15T21:04:47.766432+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:47.766432+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27594642,"num_objects":3369,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3371,"num_write_kb":26956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3378","reported_seq":3411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.762379+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:47.762379+0000","last_peered":"2024-09-15T21:04:47.762379+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:47.762379+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662982,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3377,"num_write_kb":27015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3353","reported_seq":3427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:46.059434+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:46.059434+0000","last_peered":"2024-09-15T21:04:46.059434+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:46.059434+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27458182,"num_objects":3352,"num_object_clones":0,"num_object_copies":13408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3352,"num_write_kb":26815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1066}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3432","reported_seq":3483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.761428+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:47.761428+0000","last_peered":"2024-09-15T21:04:47.761428+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:47.761428+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079372,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":27436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3403","reported_seq":3480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:46.048959+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:46.048959+0000","last_peered":"2024-09-15T21:04:46.048959+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:46.048959+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3403,"log_dups_size":0,"ondisk_log_size":3403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3401,"num_write_kb":27205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3491","reported_seq":3553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.426574+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:47.426574+0000","last_peered":"2024-09-15T21:04:47.426574+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:47.426574+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554508,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":448677544,"num_objects":54775,"num_object_clones":0,"num_object_copies":219100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54802,"num_write_kb":438298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":745459712,"data_stored":745459712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54829,"ondisk_log_size":54829,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391048,"kb_used_data":219128,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93980792,"statfs":{"total":96636764160,"available":96236331008,"internally_reserved":0,"allocated":224387072,"data_stored":224230122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":392856,"kb_used_data":217288,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93978984,"statfs":{"total":96636764160,"available":96234479616,"internally_reserved":0,"allocated":222502912,"data_stored":222345962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051595,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":165792,"kb_used_data":71696,"kb_used_omap":4,"kb_used_meta":94075,"kb_avail":94206048,"statfs":{"total":96636764160,"available":96466993152,"internally_reserved":0,"allocated":73416704,"data_stored":73259754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":96333519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,9,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402680,"kb_used_data":218344,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969160,"statfs":{"total":96636764160,"available":96224419840,"internally_reserved":0,"allocated":223584256,"data_stored":223427306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224514048,"data_stored":224514048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73453568,"data_stored":73453568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":222584832,"data_stored":222584832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224907264,"data_stored":224907264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:49.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 86 20 55490 55470 5.03842 2.65625 0.0225281 0.0309976 2024-09-15T21:04:50.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 87 20 55851 55831 5.01292 2.82031 0.0396151 0.0311542 2024-09-15T21:04:51.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 88 20 56221 56201 4.9888 2.89062 0.0605937 0.0313047 2024-09-15T21:04:52.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:52.637 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 89 20 56569 56549 4.96329 2.71875 0.0374118 0.0314632 2024-09-15T21:04:52.825 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:52.826 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:52.847 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-15T21:04:51.930255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451970728,"num_objects":55177,"num_object_clones":0,"num_object_copies":220708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55204,"num_write_kb":441514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55231,"ondisk_log_size":55231,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1376008,"kb_used_data":740356,"kb_used_omap":14,"kb_used_meta":635569,"kb_avail":376111352,"statfs":{"total":386547056640,"available":385138024448,"internally_reserved":0,"allocated":758124544,"data_stored":757496744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":650823218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,4,6,12,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32669696,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"1.f","version":"31'3350","reported_seq":3390,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.679328+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:47.679328+0000","last_peered":"2024-09-15T21:04:47.679328+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:47.679328+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3463","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.452446+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:47.452446+0000","last_peered":"2024-09-15T21:04:47.452446+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:47.452446+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3463,"log_dups_size":0,"ondisk_log_size":3463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28333324,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":27684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3422","reported_seq":3495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.048374+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:51.048374+0000","last_peered":"2024-09-15T21:04:51.048374+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:51.048374+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28007046,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3478","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.414731+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:47.414731+0000","last_peered":"2024-09-15T21:04:47.414731+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:47.414731+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456204,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3460","reported_seq":3510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.775469+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:47.775469+0000","last_peered":"2024-09-15T21:04:47.775469+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:47.775469+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3399","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.697733+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:47.697733+0000","last_peered":"2024-09-15T21:04:47.697733+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:47.697733+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3479","reported_seq":3543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.408330+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:47.408330+0000","last_peered":"2024-09-15T21:04:47.408330+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:47.408330+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28448012,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1107}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3592","reported_seq":3655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.342342+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:47.342342+0000","last_peered":"2024-09-15T21:04:47.342342+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:47.342342+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3422","reported_seq":3476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.634330+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:47.634330+0000","last_peered":"2024-09-15T21:04:47.634330+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:47.634330+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27961880,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":27329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3542","reported_seq":3618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.064870+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:51.064870+0000","last_peered":"2024-09-15T21:04:51.064870+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:51.064870+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3374","reported_seq":4559,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.766432+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:47.766432+0000","last_peered":"2024-09-15T21:04:47.766432+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:47.766432+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27594642,"num_objects":3369,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3371,"num_write_kb":26956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3378","reported_seq":3411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.762379+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:47.762379+0000","last_peered":"2024-09-15T21:04:47.762379+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:47.762379+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662982,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3377,"num_write_kb":27015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3461","reported_seq":3535,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.050238+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:51.050238+0000","last_peered":"2024-09-15T21:04:51.050238+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:51.050238+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28342918,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":27679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3432","reported_seq":3483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.761428+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:47.761428+0000","last_peered":"2024-09-15T21:04:47.761428+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:47.761428+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079372,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":27436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3488","reported_seq":3565,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:50.992846+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:50.992846+0000","last_peered":"2024-09-15T21:04:50.992846+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:50.992846+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1204}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3491","reported_seq":3553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.426574+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:47.426574+0000","last_peered":"2024-09-15T21:04:47.426574+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:47.426574+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554508,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451970728,"num_objects":55177,"num_object_clones":0,"num_object_copies":220708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55204,"num_write_kb":441514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":759562240,"data_stored":759562240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55231,"ondisk_log_size":55231,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391048,"kb_used_data":219128,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93980792,"statfs":{"total":96636764160,"available":96236331008,"internally_reserved":0,"allocated":224387072,"data_stored":224230122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403808,"kb_used_data":224076,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93968032,"statfs":{"total":96636764160,"available":96223264768,"internally_reserved":0,"allocated":229453824,"data_stored":229296874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178472,"kb_used_data":78808,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94193368,"statfs":{"total":96636764160,"available":96454008832,"internally_reserved":0,"allocated":80699392,"data_stored":80542442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":102035151},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402680,"kb_used_data":218344,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969160,"statfs":{"total":96636764160,"available":96224419840,"internally_reserved":0,"allocated":223584256,"data_stored":223427306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224514048,"data_stored":224514048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80605184,"data_stored":80605184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229535744,"data_stored":229535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224907264,"data_stored":224907264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:52.848 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:53.457 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:53.457 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:53.486 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-15T21:04:51.930255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451970728,"num_objects":55177,"num_object_clones":0,"num_object_copies":220708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55204,"num_write_kb":441514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55231,"ondisk_log_size":55231,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1376008,"kb_used_data":740356,"kb_used_omap":14,"kb_used_meta":635569,"kb_avail":376111352,"statfs":{"total":386547056640,"available":385138024448,"internally_reserved":0,"allocated":758124544,"data_stored":757496744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":650823218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,4,6,12,11,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32669696,"num_objects":3988,"num_object_clones":0,"num_object_copies":15952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3988,"num_write_kb":31904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"1.f","version":"31'3350","reported_seq":3390,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.679328+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:47.679328+0000","last_peered":"2024-09-15T21:04:47.679328+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:47.679328+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3350,"log_dups_size":0,"ondisk_log_size":3350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27443200,"num_objects":3350,"num_object_clones":0,"num_object_copies":13400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3350,"num_write_kb":26800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3463","reported_seq":3525,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.452446+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:47.452446+0000","last_peered":"2024-09-15T21:04:47.452446+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:47.452446+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3463,"log_dups_size":0,"ondisk_log_size":3463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28333324,"num_objects":3459,"num_object_clones":0,"num_object_copies":13836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":27684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3422","reported_seq":3495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.048374+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:51.048374+0000","last_peered":"2024-09-15T21:04:51.048374+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:51.048374+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28007046,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3478","reported_seq":3541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.414731+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:47.414731+0000","last_peered":"2024-09-15T21:04:47.414731+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:47.414731+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28456204,"num_objects":3474,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3476,"num_write_kb":27797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1140}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3460","reported_seq":3510,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.775469+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:47.775469+0000","last_peered":"2024-09-15T21:04:47.775469+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:47.775469+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3460,"log_dups_size":0,"ondisk_log_size":3460,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3458,"num_write_kb":27661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1146}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3399","reported_seq":3438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.697733+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:47.697733+0000","last_peered":"2024-09-15T21:04:47.697733+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:47.697733+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1083}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3479","reported_seq":3543,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.408330+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:47.408330+0000","last_peered":"2024-09-15T21:04:47.408330+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:47.408330+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3479,"log_dups_size":0,"ondisk_log_size":3479,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28448012,"num_objects":3473,"num_object_clones":0,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":27796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1107}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3592","reported_seq":3655,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.342342+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:47.342342+0000","last_peered":"2024-09-15T21:04:47.342342+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:47.342342+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29381900,"num_objects":3587,"num_object_clones":0,"num_object_copies":14348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3590,"num_write_kb":28708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1113}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3422","reported_seq":3476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.634330+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:47.634330+0000","last_peered":"2024-09-15T21:04:47.634330+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:47.634330+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27961880,"num_objects":3414,"num_object_clones":0,"num_object_copies":13656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3419,"num_write_kb":27329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1082}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3542","reported_seq":3618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.064870+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:51.064870+0000","last_peered":"2024-09-15T21:04:51.064870+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:51.064870+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3374","reported_seq":4559,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.766432+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:47.766432+0000","last_peered":"2024-09-15T21:04:47.766432+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:47.766432+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3374,"log_dups_size":0,"ondisk_log_size":3374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27594642,"num_objects":3369,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3371,"num_write_kb":26956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":568,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3378","reported_seq":3411,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.762379+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:47.762379+0000","last_peered":"2024-09-15T21:04:47.762379+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:47.762379+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27662982,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3377,"num_write_kb":27015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1121}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3461","reported_seq":3535,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.050238+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:51.050238+0000","last_peered":"2024-09-15T21:04:51.050238+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:51.050238+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28342918,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":27679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3432","reported_seq":3483,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.761428+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:47.761428+0000","last_peered":"2024-09-15T21:04:47.761428+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:47.761428+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3432,"log_dups_size":0,"ondisk_log_size":3432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28079372,"num_objects":3428,"num_object_clones":0,"num_object_copies":13712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3430,"num_write_kb":27436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1149}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3488","reported_seq":3565,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:50.992846+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:50.992846+0000","last_peered":"2024-09-15T21:04:50.992846+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:50.992846+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1204}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3491","reported_seq":3553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:47.426574+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:47.426574+0000","last_peered":"2024-09-15T21:04:47.426574+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:47.426574+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3491,"log_dups_size":0,"ondisk_log_size":3491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28554508,"num_objects":3486,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3489,"num_write_kb":27900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1171}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451970728,"num_objects":55177,"num_object_clones":0,"num_object_copies":220708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55204,"num_write_kb":441514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":4628480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":759562240,"data_stored":759562240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55231,"ondisk_log_size":55231,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":391048,"kb_used_data":219128,"kb_used_omap":4,"kb_used_meta":171899,"kb_avail":93980792,"statfs":{"total":96636764160,"available":96236331008,"internally_reserved":0,"allocated":224387072,"data_stored":224230122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":176025296},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,8,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403808,"kb_used_data":224076,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93968032,"statfs":{"total":96636764160,"available":96223264768,"internally_reserved":0,"allocated":229453824,"data_stored":229296874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178472,"kb_used_data":78808,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94193368,"statfs":{"total":96636764160,"available":96454008832,"internally_reserved":0,"allocated":80699392,"data_stored":80542442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":102035151},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402680,"kb_used_data":218344,"kb_used_omap":4,"kb_used_meta":184315,"kb_avail":93969160,"statfs":{"total":96636764160,"available":96224419840,"internally_reserved":0,"allocated":223584256,"data_stored":223427306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":188739278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":224514048,"data_stored":224514048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80605184,"data_stored":80605184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229535744,"data_stored":229535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":224907264,"data_stored":224907264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:53.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3350", 'reported_seq': 3390, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.679328+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:47.679328+0000', 'last_peered': '2024-09-15T21:04:47.679328+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:47.679328+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3350, 'log_dups_size': 0, 'ondisk_log_size': 3350, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27443200, 'num_objects': 3350, 'num_object_clones': 0, 'num_object_copies': 13400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3350, 'num_write_kb': 26800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3463", 'reported_seq': 3525, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.452446+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:47.452446+0000', 'last_peered': '2024-09-15T21:04:47.452446+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:47.452446+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3463, 'log_dups_size': 0, 'ondisk_log_size': 3463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28333324, 'num_objects': 3459, 'num_object_clones': 0, 'num_object_copies': 13836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3461, 'num_write_kb': 27684, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3422", 'reported_seq': 3495, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:51.048374+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:51.048374+0000', 'last_peered': '2024-09-15T21:04:51.048374+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:51.048374+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3422, 'log_dups_size': 0, 'ondisk_log_size': 3422, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28007046, 'num_objects': 3419, 'num_object_clones': 0, 'num_object_copies': 13676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3421, 'num_write_kb': 27358, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1161}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3478", 'reported_seq': 3541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.414731+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:47.414731+0000', 'last_peered': '2024-09-15T21:04:47.414731+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:47.414731+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3478, 'log_dups_size': 0, 'ondisk_log_size': 3478, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28456204, 'num_objects': 3474, 'num_object_clones': 0, 'num_object_copies': 13896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3476, 'num_write_kb': 27797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1140}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:53.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3460", 'reported_seq': 3510, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.775469+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:47.775469+0000', 'last_peered': '2024-09-15T21:04:47.775469+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:47.775469+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3460, 'log_dups_size': 0, 'ondisk_log_size': 3460, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28316940, 'num_objects': 3457, 'num_object_clones': 0, 'num_object_copies': 13828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3458, 'num_write_kb': 27661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1146}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3399", 'reported_seq': 3438, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.697733+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:47.697733+0000', 'last_peered': '2024-09-15T21:04:47.697733+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:47.697733+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3399, 'log_dups_size': 0, 'ondisk_log_size': 3399, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27844608, 'num_objects': 3399, 'num_object_clones': 0, 'num_object_copies': 13596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3399, 'num_write_kb': 27192, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1083}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3479", 'reported_seq': 3543, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.408330+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:47.408330+0000', 'last_peered': '2024-09-15T21:04:47.408330+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:47.408330+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3479, 'log_dups_size': 0, 'ondisk_log_size': 3479, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28448012, 'num_objects': 3473, 'num_object_clones': 0, 'num_object_copies': 13892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3477, 'num_write_kb': 27796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1107}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3592", 'reported_seq': 3655, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.342342+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:47.342342+0000', 'last_peered': '2024-09-15T21:04:47.342342+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:47.342342+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3592, 'log_dups_size': 0, 'ondisk_log_size': 3592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29381900, 'num_objects': 3587, 'num_object_clones': 0, 'num_object_copies': 14348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3587, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3590, 'num_write_kb': 28708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1113}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3422", 'reported_seq': 3476, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.634330+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:47.634330+0000', 'last_peered': '2024-09-15T21:04:47.634330+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:47.634330+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3422, 'log_dups_size': 0, 'ondisk_log_size': 3422, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27961880, 'num_objects': 3414, 'num_object_clones': 0, 'num_object_copies': 13656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3419, 'num_write_kb': 27329, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1082}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3542", 'reported_seq': 3618, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:51.064870+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:51.064870+0000', 'last_peered': '2024-09-15T21:04:51.064870+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:51.064870+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3542, 'log_dups_size': 0, 'ondisk_log_size': 3542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28996876, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3540, 'num_write_kb': 28318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1205}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:53.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3374", 'reported_seq': 4559, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.766432+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:47.766432+0000', 'last_peered': '2024-09-15T21:04:47.766432+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:47.766432+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3374, 'log_dups_size': 0, 'ondisk_log_size': 3374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27594642, 'num_objects': 3369, 'num_object_clones': 0, 'num_object_copies': 13476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1695, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3371, 'num_write_kb': 26956, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 568, 'num_bytes_recovered': 4628480, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1674}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3378", 'reported_seq': 3411, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.762379+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:47.762379+0000', 'last_peered': '2024-09-15T21:04:47.762379+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:47.762379+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3378, 'log_dups_size': 0, 'ondisk_log_size': 3378, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27662982, 'num_objects': 3377, 'num_object_clones': 0, 'num_object_copies': 13508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3377, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3377, 'num_write_kb': 27015, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1121}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3461", 'reported_seq': 3535, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:51.050238+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:51.050238+0000', 'last_peered': '2024-09-15T21:04:51.050238+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:51.050238+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3461, 'log_dups_size': 0, 'ondisk_log_size': 3461, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28342918, 'num_objects': 3460, 'num_object_clones': 0, 'num_object_copies': 13840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3460, 'num_write_kb': 27679, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1174}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3432", 'reported_seq': 3483, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.761428+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:47.761428+0000', 'last_peered': '2024-09-15T21:04:47.761428+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:47.761428+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3432, 'log_dups_size': 0, 'ondisk_log_size': 3432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28079372, 'num_objects': 3428, 'num_object_clones': 0, 'num_object_copies': 13712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3430, 'num_write_kb': 27436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3488", 'reported_seq': 3565, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:50.992846+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:50.992846+0000', 'last_peered': '2024-09-15T21:04:50.992846+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:50.992846+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3488, 'log_dups_size': 0, 'ondisk_log_size': 3488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28546316, 'num_objects': 3485, 'num_object_clones': 0, 'num_object_copies': 13940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3486, 'num_write_kb': 27885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1204}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:53.490 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3491", 'reported_seq': 3553, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:47.426574+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:47.426574+0000', 'last_peered': '2024-09-15T21:04:47.426574+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:47.426574+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3491, 'log_dups_size': 0, 'ondisk_log_size': 3491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28554508, 'num_objects': 3486, 'num_object_clones': 0, 'num_object_copies': 13944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3489, 'num_write_kb': 27900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1171}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:53.491 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:53.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 90 20 56867 56847 4.93401 2.32812 0.0509637 0.03165 2024-09-15T21:04:54.206 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:54.206 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36518/220708 objects degraded (16.546%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":159,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55177,"data_bytes":451970728,"bytes_used":1409032192,"bytes_avail":385138024448,"bytes_total":386547056640,"degraded_objects":36518,"degraded_total":220708,"degraded_ratio":0.16545843376769306,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":83262,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":999424,"num_keys_recovered":0,"write_bytes_sec":2721733,"read_op_per_sec":0,"write_op_per_sec":332},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-15T21:04:13.920325+0000","services":{"osd":{"daemons":{"summary":"","1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-15T21:04:54.240 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:54.240 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:54.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 91 20 57185 57165 4.90709 2.48438 0.0454112 0.0318252 2024-09-15T21:04:54.844 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:54.845 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:54.873 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-09-15T21:04:53.931477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462620328,"num_objects":56477,"num_object_clones":0,"num_object_copies":225908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56504,"num_write_kb":451914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":5079040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56531,"ondisk_log_size":56531,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1381364,"kb_used_data":754032,"kb_used_omap":14,"kb_used_meta":627249,"kb_avail":376105996,"statfs":{"total":386547056640,"available":385132539904,"internally_reserved":0,"allocated":772128768,"data_stored":771500968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":642303538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,3,2,8,9,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39682048,"num_objects":4844,"num_object_clones":0,"num_object_copies":19376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003887"},"pg_stats":[{"pgid":"1.f","version":"31'3453","reported_seq":3493,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.808193+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:52.808193+0000","last_peered":"2024-09-15T21:04:52.808193+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:52.808193+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28286976,"num_objects":3453,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3453,"num_write_kb":27624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1177}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3583","reported_seq":3645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.486242+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:52.486242+0000","last_peered":"2024-09-15T21:04:52.486242+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:52.486242+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29316364,"num_objects":3579,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3581,"num_write_kb":28644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1199}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3422","reported_seq":3495,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.048374+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:51.048374+0000","last_peered":"2024-09-15T21:04:51.048374+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:51.048374+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3422,"log_dups_size":0,"ondisk_log_size":3422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28007046,"num_objects":3419,"num_object_clones":0,"num_object_copies":13676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3421,"num_write_kb":27358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1161}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3580","reported_seq":3643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.562284+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:52.562284+0000","last_peered":"2024-09-15T21:04:52.562284+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:52.562284+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29291788,"num_objects":3576,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3578,"num_write_kb":28613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3567","reported_seq":3617,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.785544+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:52.785544+0000","last_peered":"2024-09-15T21:04:52.785544+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:52.785544+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3567,"log_dups_size":0,"ondisk_log_size":3567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29193484,"num_objects":3564,"num_object_clones":0,"num_object_copies":14256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3565,"num_write_kb":28517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3515","reported_seq":3554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.731705+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:52.731705+0000","last_peered":"2024-09-15T21:04:52.731705+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:52.731705+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28794880,"num_objects":3515,"num_object_clones":0,"num_object_copies":14060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":28120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3584","reported_seq":3648,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.528856+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:52.528856+0000","last_peered":"2024-09-15T21:04:52.528856+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:52.528856+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3584,"log_dups_size":0,"ondisk_log_size":3584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308172,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3582,"num_write_kb":28636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3695","reported_seq":3756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.526027+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:52.526027+0000","last_peered":"2024-09-15T21:04:52.526027+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:52.526027+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3695,"log_dups_size":0,"ondisk_log_size":3695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30225676,"num_objects":3690,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":29532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1216}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3525","reported_seq":3579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.781661+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:52.781661+0000","last_peered":"2024-09-15T21:04:52.781661+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:52.781661+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28805656,"num_objects":3517,"num_object_clones":0,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":28153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3542","reported_seq":3618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.064870+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:51.064870+0000","last_peered":"2024-09-15T21:04:51.064870+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:51.064870+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3542,"log_dups_size":0,"ondisk_log_size":3542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3540,"num_write_kb":28318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3488","reported_seq":4783,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.840129+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:52.840129+0000","last_peered":"2024-09-15T21:04:52.840129+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:52.840129+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28528530,"num_objects":3483,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1640,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3485,"num_write_kb":27868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":623,"num_bytes_recovered":5079040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1843}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3496","reported_seq":3529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.776130+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:52.776130+0000","last_peered":"2024-09-15T21:04:52.776130+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:52.776130+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28629638,"num_objects":3495,"num_object_clones":0,"num_object_copies":13980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3495,"num_write_kb":27959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1239}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3461","reported_seq":3535,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:51.050238+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:51.050238+0000","last_peered":"2024-09-15T21:04:51.050238+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:51.050238+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28342918,"num_objects":3460,"num_object_clones":0,"num_object_copies":13840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":27679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3538","reported_seq":3589,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.803731+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:52.803731+0000","last_peered":"2024-09-15T21:04:52.803731+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:52.803731+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28947724,"num_objects":3534,"num_object_clones":0,"num_object_copies":14136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":28284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3488","reported_seq":3565,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:50.992846+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:50.992846+0000","last_peered":"2024-09-15T21:04:50.992846+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:50.992846+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3488,"log_dups_size":0,"ondisk_log_size":3488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3486,"num_write_kb":27885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1204}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3594","reported_seq":3657,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:52.525581+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:52.525581+0000","last_peered":"2024-09-15T21:04:52.525581+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:52.525581+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3594,"log_dups_size":0,"ondisk_log_size":3594,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29398284,"num_objects":3589,"num_object_clones":0,"num_object_copies":14356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3592,"num_write_kb":28724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462620328,"num_objects":56477,"num_object_clones":0,"num_object_copies":225908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56504,"num_write_kb":451914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":5079040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":773771264,"data_stored":773771264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56531,"ondisk_log_size":56531,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":402216,"kb_used_data":226136,"kb_used_omap":4,"kb_used_meta":176059,"kb_avail":93969624,"statfs":{"total":96636764160,"available":96224894976,"internally_reserved":0,"allocated":231563264,"data_stored":231406314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":180285136},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,1,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":403808,"kb_used_data":224076,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93968032,"statfs":{"total":96636764160,"available":96223264768,"internally_reserved":0,"allocated":229453824,"data_stored":229296874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051596,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178472,"kb_used_data":78808,"kb_used_omap":4,"kb_used_meta":99643,"kb_avail":94193368,"statfs":{"total":96636764160,"available":96454008832,"internally_reserved":0,"allocated":80699392,"data_stored":80542442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":102035151},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396868,"kb_used_data":225012,"kb_used_omap":4,"kb_used_meta":171835,"kb_avail":93974972,"statfs":{"total":96636764160,"available":96230371328,"internally_reserved":0,"allocated":230412288,"data_stored":230255338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":175959758},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,0,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231649280,"data_stored":231649280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":80605184,"data_stored":80605184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":229535744,"data_stored":229535744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231981056,"data_stored":231981056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:55.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 92 20 57531 57511 4.88313 2.70312 0.0635696 0.031984 2024-09-15T21:04:56.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 93 20 57890 57870 4.86078 2.80469 0.0684214 0.0321334 2024-09-15T21:04:57.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 94 20 58246 58226 4.83865 2.78125 0.0257868 0.032274 2024-09-15T21:04:57.875 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:58.477 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:58.478 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:58.504 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-15T21:04:57.932710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1448480,"kb_used_data":782304,"kb_used_omap":14,"kb_used_meta":666097,"kb_avail":376038880,"statfs":{"total":386547056640,"available":385063813120,"internally_reserved":0,"allocated":801079296,"data_stored":800451496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":682083890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,21,11,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42467328,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5184,"num_write_kb":41472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003777"},"pg_stats":[{"pgid":"1.f","version":"31'3585","reported_seq":3624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.848762+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:57.848762+0000","last_peered":"2024-09-15T21:04:57.848762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:57.848762+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29368320,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3585,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3681","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.588922+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:57.588922+0000","last_peered":"2024-09-15T21:04:57.588922+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:57.588922+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119180,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3540","reported_seq":3614,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.151482+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:56.151482+0000","last_peered":"2024-09-15T21:04:56.151482+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:56.151482+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28973702,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3686","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.534141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:57.534141+0000","last_peered":"2024-09-15T21:04:57.534141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:57.534141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3679","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.843632+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:57.843632+0000","last_peered":"2024-09-15T21:04:57.843632+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:57.843632+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3679,"log_dups_size":0,"ondisk_log_size":3679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30110988,"num_objects":3676,"num_object_clones":0,"num_object_copies":14704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3677,"num_write_kb":29413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3614","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.877802+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:57.877802+0000","last_peered":"2024-09-15T21:04:57.877802+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:57.877802+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3691","reported_seq":3754,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.557929+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:57.557929+0000","last_peered":"2024-09-15T21:04:57.557929+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:57.557929+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30184716,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3781","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.449523+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:57.449523+0000","last_peered":"2024-09-15T21:04:57.449523+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:57.449523+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930188,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3638","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.850504+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:57.850504+0000","last_peered":"2024-09-15T21:04:57.850504+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:57.850504+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29731352,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3651","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.086973+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:56.086973+0000","last_peered":"2024-09-15T21:04:56.086973+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:56.086973+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889804,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3600","reported_seq":5011,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.868883+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:57.868883+0000","last_peered":"2024-09-15T21:04:57.868883+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:57.868883+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446034,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2013}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3601","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.665495+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:57.665495+0000","last_peered":"2024-09-15T21:04:57.665495+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:57.665495+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29489798,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":28799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3561","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.100176+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:56.100176+0000","last_peered":"2024-09-15T21:04:56.100176+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:56.100176+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3560,"num_write_kb":28479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3642","reported_seq":3693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.849854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:57.849854+0000","last_peered":"2024-09-15T21:04:57.849854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:57.849854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799692,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3640,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3591","reported_seq":3668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.094097+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:56.094097+0000","last_peered":"2024-09-15T21:04:56.094097+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:56.094097+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390092,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3687","reported_seq":3750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.556578+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:57.556578+0000","last_peered":"2024-09-15T21:04:57.556578+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:57.556578+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":29468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801624064,"data_stored":801624064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417212,"kb_used_data":232812,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93954628,"statfs":{"total":96636764160,"available":96209539072,"internally_reserved":0,"allocated":238399488,"data_stored":238242538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414500,"kb_used_data":230612,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93957340,"statfs":{"total":96636764160,"available":96212316160,"internally_reserved":0,"allocated":236146688,"data_stored":235989738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,6,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":207692,"kb_used_data":85820,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94164148,"statfs":{"total":96636764160,"available":96424087552,"internally_reserved":0,"allocated":87879680,"data_stored":87722730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409076,"kb_used_data":233060,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93962764,"statfs":{"total":96636764160,"available":96217870336,"internally_reserved":0,"allocated":238653440,"data_stored":238496490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87707648,"data_stored":87707648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236392448,"data_stored":236392448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238989312,"data_stored":238989312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:58.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:04:58.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 95 20 58578 58558 4.81502 2.59375 0.0296221 0.0324388 2024-09-15T21:04:59.136 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:59.136 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:04:59.157 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2024-09-15T21:04:57.932710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1448480,"kb_used_data":782304,"kb_used_omap":14,"kb_used_meta":666097,"kb_avail":376038880,"statfs":{"total":386547056640,"available":385063813120,"internally_reserved":0,"allocated":801079296,"data_stored":800451496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":682083890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,21,11,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42467328,"num_objects":5184,"num_object_clones":0,"num_object_copies":20736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5184,"num_write_kb":41472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003777"},"pg_stats":[{"pgid":"1.f","version":"31'3585","reported_seq":3624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.848762+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:57.848762+0000","last_peered":"2024-09-15T21:04:57.848762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:57.848762+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29368320,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3585,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3681","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.588922+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:57.588922+0000","last_peered":"2024-09-15T21:04:57.588922+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:57.588922+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119180,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3540","reported_seq":3614,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.151482+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:56.151482+0000","last_peered":"2024-09-15T21:04:56.151482+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:56.151482+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28973702,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3686","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.534141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:57.534141+0000","last_peered":"2024-09-15T21:04:57.534141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:57.534141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3679","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.843632+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:57.843632+0000","last_peered":"2024-09-15T21:04:57.843632+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:57.843632+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3679,"log_dups_size":0,"ondisk_log_size":3679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30110988,"num_objects":3676,"num_object_clones":0,"num_object_copies":14704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3677,"num_write_kb":29413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3614","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.877802+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:57.877802+0000","last_peered":"2024-09-15T21:04:57.877802+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:57.877802+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3691","reported_seq":3754,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.557929+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:57.557929+0000","last_peered":"2024-09-15T21:04:57.557929+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:57.557929+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30184716,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3781","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.449523+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:57.449523+0000","last_peered":"2024-09-15T21:04:57.449523+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:57.449523+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930188,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3638","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.850504+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:57.850504+0000","last_peered":"2024-09-15T21:04:57.850504+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:57.850504+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29731352,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3651","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.086973+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:56.086973+0000","last_peered":"2024-09-15T21:04:56.086973+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:56.086973+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889804,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3600","reported_seq":5011,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.868883+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:57.868883+0000","last_peered":"2024-09-15T21:04:57.868883+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:57.868883+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446034,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2013}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3601","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.665495+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:57.665495+0000","last_peered":"2024-09-15T21:04:57.665495+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:57.665495+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29489798,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":28799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3561","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.100176+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:56.100176+0000","last_peered":"2024-09-15T21:04:56.100176+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:56.100176+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3560,"num_write_kb":28479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3642","reported_seq":3693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.849854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:57.849854+0000","last_peered":"2024-09-15T21:04:57.849854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:57.849854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799692,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3640,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3591","reported_seq":3668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.094097+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:56.094097+0000","last_peered":"2024-09-15T21:04:56.094097+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:56.094097+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390092,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3687","reported_seq":3750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.556578+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:57.556578+0000","last_peered":"2024-09-15T21:04:57.556578+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:57.556578+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":29468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801624064,"data_stored":801624064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417212,"kb_used_data":232812,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93954628,"statfs":{"total":96636764160,"available":96209539072,"internally_reserved":0,"allocated":238399488,"data_stored":238242538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414500,"kb_used_data":230612,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93957340,"statfs":{"total":96636764160,"available":96212316160,"internally_reserved":0,"allocated":236146688,"data_stored":235989738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,6,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":207692,"kb_used_data":85820,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94164148,"statfs":{"total":96636764160,"available":96424087552,"internally_reserved":0,"allocated":87879680,"data_stored":87722730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409076,"kb_used_data":233060,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93962764,"statfs":{"total":96636764160,"available":96217870336,"internally_reserved":0,"allocated":238653440,"data_stored":238496490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87707648,"data_stored":87707648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236392448,"data_stored":236392448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238989312,"data_stored":238989312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3585", 'reported_seq': 3624, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.848762+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:04:57.848762+0000', 'last_peered': '2024-09-15T21:04:57.848762+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:04:57.848762+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3585, 'log_dups_size': 0, 'ondisk_log_size': 3585, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29368320, 'num_objects': 3585, 'num_object_clones': 0, 'num_object_copies': 14340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3585, 'num_write_kb': 28680, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3681", 'reported_seq': 3743, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.588922+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:04:57.588922+0000', 'last_peered': '2024-09-15T21:04:57.588922+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:04:57.588922+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3681, 'log_dups_size': 0, 'ondisk_log_size': 3681, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30119180, 'num_objects': 3677, 'num_object_clones': 0, 'num_object_copies': 14708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3679, 'num_write_kb': 29428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1297}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3540", 'reported_seq': 3614, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:56.151482+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:04:56.151482+0000', 'last_peered': '2024-09-15T21:04:56.151482+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:04:56.151482+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3540, 'log_dups_size': 0, 'ondisk_log_size': 3540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28973702, 'num_objects': 3537, 'num_object_clones': 0, 'num_object_copies': 14148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3537, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3539, 'num_write_kb': 28302, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1279}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:59.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3686", 'reported_seq': 3749, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.534141+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:04:57.534141+0000', 'last_peered': '2024-09-15T21:04:57.534141+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:04:57.534141+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3686, 'log_dups_size': 0, 'ondisk_log_size': 3686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30160140, 'num_objects': 3682, 'num_object_clones': 0, 'num_object_copies': 14728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3684, 'num_write_kb': 29461, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3679", 'reported_seq': 3729, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.843632+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:04:57.843632+0000', 'last_peered': '2024-09-15T21:04:57.843632+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:04:57.843632+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3679, 'log_dups_size': 0, 'ondisk_log_size': 3679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30110988, 'num_objects': 3676, 'num_object_clones': 0, 'num_object_copies': 14704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3677, 'num_write_kb': 29413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1365}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3614", 'reported_seq': 3652, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.877802+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:04:57.877802+0000', 'last_peered': '2024-09-15T21:04:57.877802+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:04:57.877802+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3614, 'log_dups_size': 0, 'ondisk_log_size': 3614, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29605888, 'num_objects': 3614, 'num_object_clones': 0, 'num_object_copies': 14456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3614, 'num_write_kb': 28912, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1298}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3691", 'reported_seq': 3754, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.557929+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:04:57.557929+0000', 'last_peered': '2024-09-15T21:04:57.557929+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:04:57.557929+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3691, 'log_dups_size': 0, 'ondisk_log_size': 3691, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30184716, 'num_objects': 3685, 'num_object_clones': 0, 'num_object_copies': 14740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3689, 'num_write_kb': 29492, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1319}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3781", 'reported_seq': 3844, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.449523+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:04:57.449523+0000', 'last_peered': '2024-09-15T21:04:57.449523+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:04:57.449523+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3781, 'log_dups_size': 0, 'ondisk_log_size': 3781, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30930188, 'num_objects': 3776, 'num_object_clones': 0, 'num_object_copies': 15104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3779, 'num_write_kb': 30220, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:04:59.159 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3638", 'reported_seq': 3689, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.850504+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:04:57.850504+0000', 'last_peered': '2024-09-15T21:04:57.850504+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:04:57.850504+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3638, 'log_dups_size': 0, 'ondisk_log_size': 3638, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29731352, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3635, 'num_write_kb': 29057, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1298}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3651", 'reported_seq': 3727, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:56.086973+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:04:56.086973+0000', 'last_peered': '2024-09-15T21:04:56.086973+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:04:56.086973+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3651, 'log_dups_size': 0, 'ondisk_log_size': 3651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29889804, 'num_objects': 3649, 'num_object_clones': 0, 'num_object_copies': 14596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3649, 'num_write_kb': 29190, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1314}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3600", 'reported_seq': 5011, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.868883+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:04:57.868883+0000', 'last_peered': '2024-09-15T21:04:57.868883+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:04:57.868883+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3600, 'log_dups_size': 0, 'ondisk_log_size': 3600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29446034, 'num_objects': 3595, 'num_object_clones': 0, 'num_object_copies': 14380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1582, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3597, 'num_write_kb': 28764, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 681, 'num_bytes_recovered': 5554176, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2013}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3601", 'reported_seq': 3635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.665495+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:04:57.665495+0000', 'last_peered': '2024-09-15T21:04:57.665495+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:04:57.665495+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3601, 'log_dups_size': 0, 'ondisk_log_size': 3601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29489798, 'num_objects': 3600, 'num_object_clones': 0, 'num_object_copies': 14400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3600, 'num_write_kb': 28799, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1344}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3561", 'reported_seq': 3635, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:56.100176+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:04:56.100176+0000', 'last_peered': '2024-09-15T21:04:56.100176+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:04:56.100176+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3561, 'log_dups_size': 0, 'ondisk_log_size': 3561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29162118, 'num_objects': 3560, 'num_object_clones': 0, 'num_object_copies': 14240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3560, 'num_write_kb': 28479, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3642", 'reported_seq': 3693, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.849854+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:04:57.849854+0000', 'last_peered': '2024-09-15T21:04:57.849854+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:04:57.849854+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3642, 'log_dups_size': 0, 'ondisk_log_size': 3642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29799692, 'num_objects': 3638, 'num_object_clones': 0, 'num_object_copies': 14552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3638, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3640, 'num_write_kb': 29116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1359}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:04:59.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:04:59.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3591", 'reported_seq': 3668, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:56.094097+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:04:56.094097+0000', 'last_peered': '2024-09-15T21:04:56.094097+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:04:56.094097+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3591, 'log_dups_size': 0, 'ondisk_log_size': 3591, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29390092, 'num_objects': 3588, 'num_object_clones': 0, 'num_object_copies': 14352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3589, 'num_write_kb': 28709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:04:59.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:04:59.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3687", 'reported_seq': 3750, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:04:57.556578+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:04:57.556578+0000', 'last_peered': '2024-09-15T21:04:57.556578+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:04:57.556578+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3687, 'log_dups_size': 0, 'ondisk_log_size': 3687, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30160140, 'num_objects': 3682, 'num_object_clones': 0, 'num_object_copies': 14728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3685, 'num_write_kb': 29468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:04:59.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:04:59.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 96 20 58888 58868 4.79009 2.42188 0.0607455 0.0326036 2024-09-15T21:04:59.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:04:59.930 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36405/232696 objects degraded (15.645%), 16 pgs degraded","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":165,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58174,"data_bytes":476522152,"bytes_used":1483243520,"bytes_avail":385063813120,"bytes_total":386547056640,"degraded_objects":36405,"degraded_total":232696,"degraded_ratio":0.15644875717674561,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116699,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"write_bytes_sec":3537830,"read_op_per_sec":0,"write_op_per_sec":431},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:04:59.957 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:04:59.957 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:00.593 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:00.594 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:00.614 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-15T21:04:59.933570+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1448480,"kb_used_data":782304,"kb_used_omap":14,"kb_used_meta":666097,"kb_avail":376038880,"statfs":{"total":386547056640,"available":385063813120,"internally_reserved":0,"allocated":801079296,"data_stored":800451496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":682083890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,21,11,8,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27844608,"num_objects":3399,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":27192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004136"},"pg_stats":[{"pgid":"1.f","version":"31'3585","reported_seq":3624,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.848762+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:04:57.848762+0000","last_peered":"2024-09-15T21:04:57.848762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:04:57.848762+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29368320,"num_objects":3585,"num_object_clones":0,"num_object_copies":14340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3585,"num_write_kb":28680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3681","reported_seq":3743,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.588922+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:04:57.588922+0000","last_peered":"2024-09-15T21:04:57.588922+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:04:57.588922+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3681,"log_dups_size":0,"ondisk_log_size":3681,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30119180,"num_objects":3677,"num_object_clones":0,"num_object_copies":14708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":29428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1297}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3540","reported_seq":3614,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.151482+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:04:56.151482+0000","last_peered":"2024-09-15T21:04:56.151482+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:04:56.151482+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28973702,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3539,"num_write_kb":28302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3686","reported_seq":3749,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.534141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:04:57.534141+0000","last_peered":"2024-09-15T21:04:57.534141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:04:57.534141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":29461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3679","reported_seq":3729,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.843632+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:04:57.843632+0000","last_peered":"2024-09-15T21:04:57.843632+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:04:57.843632+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3679,"log_dups_size":0,"ondisk_log_size":3679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30110988,"num_objects":3676,"num_object_clones":0,"num_object_copies":14704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3677,"num_write_kb":29413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1365}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3614","reported_seq":3652,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.877802+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:04:57.877802+0000","last_peered":"2024-09-15T21:04:57.877802+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:04:57.877802+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29605888,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":28912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3691","reported_seq":3754,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.557929+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:04:57.557929+0000","last_peered":"2024-09-15T21:04:57.557929+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:04:57.557929+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30184716,"num_objects":3685,"num_object_clones":0,"num_object_copies":14740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":29492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3781","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.449523+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:04:57.449523+0000","last_peered":"2024-09-15T21:04:57.449523+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:04:57.449523+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30930188,"num_objects":3776,"num_object_clones":0,"num_object_copies":15104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3779,"num_write_kb":30220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1302}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3638","reported_seq":3689,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.850504+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:04:57.850504+0000","last_peered":"2024-09-15T21:04:57.850504+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:04:57.850504+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29731352,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3635,"num_write_kb":29057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1298}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3651","reported_seq":3727,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.086973+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:04:56.086973+0000","last_peered":"2024-09-15T21:04:56.086973+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:04:56.086973+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3651,"log_dups_size":0,"ondisk_log_size":3651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29889804,"num_objects":3649,"num_object_clones":0,"num_object_copies":14596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3600","reported_seq":5011,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.868883+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:04:57.868883+0000","last_peered":"2024-09-15T21:04:57.868883+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:04:57.868883+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29446034,"num_objects":3595,"num_object_clones":0,"num_object_copies":14380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1582,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3597,"num_write_kb":28764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2013}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3601","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.665495+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:04:57.665495+0000","last_peered":"2024-09-15T21:04:57.665495+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:04:57.665495+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29489798,"num_objects":3600,"num_object_clones":0,"num_object_copies":14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":28799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1344}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3561","reported_seq":3635,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.100176+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:04:56.100176+0000","last_peered":"2024-09-15T21:04:56.100176+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:04:56.100176+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3561,"log_dups_size":0,"ondisk_log_size":3561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3560,"num_write_kb":28479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3642","reported_seq":3693,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.849854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:04:57.849854+0000","last_peered":"2024-09-15T21:04:57.849854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:04:57.849854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3642,"log_dups_size":0,"ondisk_log_size":3642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29799692,"num_objects":3638,"num_object_clones":0,"num_object_copies":14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3640,"num_write_kb":29116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1359}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3591","reported_seq":3668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:56.094097+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:04:56.094097+0000","last_peered":"2024-09-15T21:04:56.094097+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:04:56.094097+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390092,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3589,"num_write_kb":28709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3687","reported_seq":3750,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:04:57.556578+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:04:57.556578+0000","last_peered":"2024-09-15T21:04:57.556578+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:04:57.556578+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30160140,"num_objects":3682,"num_object_clones":0,"num_object_copies":14728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3685,"num_write_kb":29468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1367}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":476522152,"num_objects":58174,"num_object_clones":0,"num_object_copies":232696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36405,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58201,"num_write_kb":465490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":5554176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":801624064,"data_stored":801624064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58228,"ondisk_log_size":58228,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":417212,"kb_used_data":232812,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93954628,"statfs":{"total":96636764160,"available":96209539072,"internally_reserved":0,"allocated":238399488,"data_stored":238242538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":188804816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,9,2,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":414500,"kb_used_data":230612,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93957340,"statfs":{"total":96636764160,"available":96212316160,"internally_reserved":0,"allocated":236146688,"data_stored":235989738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,6,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051597,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":207692,"kb_used_data":85820,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94164148,"statfs":{"total":96636764160,"available":96424087552,"internally_reserved":0,"allocated":87879680,"data_stored":87722730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409076,"kb_used_data":233060,"kb_used_omap":4,"kb_used_meta":175995,"kb_avail":93962764,"statfs":{"total":96636764160,"available":96217870336,"internally_reserved":0,"allocated":238653440,"data_stored":238496490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":180219598},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":238534656,"data_stored":238534656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87707648,"data_stored":87707648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":236392448,"data_stored":236392448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":238989312,"data_stored":238989312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:00.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 97 20 59202 59182 4.766 2.45312 0.0836288 0.0327711 2024-09-15T21:05:01.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 98 20 59561 59541 4.74598 2.80469 0.0272608 0.032905 2024-09-15T21:05:02.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 99 20 59910 59890 4.72558 2.72656 0.04101 0.033051 2024-09-15T21:05:03.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:03.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:05:03.636759+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0331996 2024-09-15T21:05:03.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:05:03.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 100 20 60247 60227 4.70466 2.63281 0.0314018 0.0331996 2024-09-15T21:05:04.228 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:04.229 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:04.255 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-15T21:05:03.935325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1529600,"kb_used_data":809276,"kb_used_omap":14,"kb_used_meta":720241,"kb_avail":375957760,"statfs":{"total":386547056640,"available":384980746240,"internally_reserved":0,"allocated":828698624,"data_stored":828070824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":737527346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,11,14,4,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38313984,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005069"},"pg_stats":[{"pgid":"1.f","version":"31'3688","reported_seq":3728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.832618+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:02.832618+0000","last_peered":"2024-09-15T21:05:02.832618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:02.832618+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3787","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.530101+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:02.530101+0000","last_peered":"2024-09-15T21:05:02.530101+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:02.530101+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987532,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":30276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3650","reported_seq":3724,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.159429+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:01.159429+0000","last_peered":"2024-09-15T21:05:01.159429+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:01.159429+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874822,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3788","reported_seq":3851,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.526144+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:02.526144+0000","last_peered":"2024-09-15T21:05:02.526144+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:02.526144+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3788,"log_dups_size":0,"ondisk_log_size":3788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3794","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.919116+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:02.919116+0000","last_peered":"2024-09-15T21:05:02.919116+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:02.919116+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053068,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3717","reported_seq":3756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.939130+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:02.939130+0000","last_peered":"2024-09-15T21:05:02.939130+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:02.939130+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3795","reported_seq":3859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.549463+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:02.549463+0000","last_peered":"2024-09-15T21:05:02.549463+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:02.549463+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31036684,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1423}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3883","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.531990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:02.531990+0000","last_peered":"2024-09-15T21:05:02.531990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:02.531990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31765772,"num_objects":3878,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3881,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3749","reported_seq":3803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.829531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:02.829531+0000","last_peered":"2024-09-15T21:05:02.829531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:02.829531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640664,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3753","reported_seq":3829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.166360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:01.166360+0000","last_peered":"2024-09-15T21:05:01.166360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:01.166360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725388,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3711","reported_seq":5238,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.983337+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:02.983337+0000","last_peered":"2024-09-15T21:05:02.983337+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:02.983337+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355346,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3721","reported_seq":3755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.951311+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:02.951311+0000","last_peered":"2024-09-15T21:05:02.951311+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:02.951311+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472838,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3663","reported_seq":3737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.069301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:01.069301+0000","last_peered":"2024-09-15T21:05:01.069301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:01.069301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29997702,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":29295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3737","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.830201+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:02.830201+0000","last_peered":"2024-09-15T21:05:02.830201+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:02.830201+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577932,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":29876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3690","reported_seq":3767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.179164+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:01.179164+0000","last_peered":"2024-09-15T21:05:01.179164+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:01.179164+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30201100,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3782","reported_seq":3845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.513665+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:02.513665+0000","last_peered":"2024-09-15T21:05:02.513665+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:02.513665+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938380,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":30228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829489152,"data_stored":829489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446028,"kb_used_data":239420,"kb_used_omap":4,"kb_used_meta":206587,"kb_avail":93925812,"statfs":{"total":96636764160,"available":96180031488,"internally_reserved":0,"allocated":245166080,"data_stored":245009130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":211545808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,4,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426868,"kb_used_data":237412,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93944972,"statfs":{"total":96636764160,"available":96199651328,"internally_reserved":0,"allocated":243109888,"data_stored":242952938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":218832,"kb_used_data":92796,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94153008,"statfs":{"total":96636764160,"available":96412680192,"internally_reserved":0,"allocated":95023104,"data_stored":94866154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437872,"kb_used_data":239648,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93933968,"statfs":{"total":96636764160,"available":96188383232,"internally_reserved":0,"allocated":245399552,"data_stored":245242602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":202960590},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245350400,"data_stored":245350400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94924800,"data_stored":94924800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243326976,"data_stored":243326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245886976,"data_stored":245886976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:04.256 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:04.638 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 101 20 60549 60529 4.68143 2.35938 0.0611488 0.033354 2024-09-15T21:05:04.886 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:04.887 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:04.923 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-15T21:05:03.935325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1529600,"kb_used_data":809276,"kb_used_omap":14,"kb_used_meta":720241,"kb_avail":375957760,"statfs":{"total":386547056640,"available":384980746240,"internally_reserved":0,"allocated":828698624,"data_stored":828070824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":737527346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,11,14,4,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":38313984,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005069"},"pg_stats":[{"pgid":"1.f","version":"31'3688","reported_seq":3728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.832618+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:02.832618+0000","last_peered":"2024-09-15T21:05:02.832618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:02.832618+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3787","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.530101+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:02.530101+0000","last_peered":"2024-09-15T21:05:02.530101+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:02.530101+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987532,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":30276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3650","reported_seq":3724,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.159429+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:01.159429+0000","last_peered":"2024-09-15T21:05:01.159429+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:01.159429+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874822,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3788","reported_seq":3851,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.526144+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:02.526144+0000","last_peered":"2024-09-15T21:05:02.526144+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:02.526144+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3788,"log_dups_size":0,"ondisk_log_size":3788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3794","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.919116+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:02.919116+0000","last_peered":"2024-09-15T21:05:02.919116+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:02.919116+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053068,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3717","reported_seq":3756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.939130+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:02.939130+0000","last_peered":"2024-09-15T21:05:02.939130+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:02.939130+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3795","reported_seq":3859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.549463+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:02.549463+0000","last_peered":"2024-09-15T21:05:02.549463+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:02.549463+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31036684,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1423}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3883","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.531990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:02.531990+0000","last_peered":"2024-09-15T21:05:02.531990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:02.531990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31765772,"num_objects":3878,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3881,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3749","reported_seq":3803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.829531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:02.829531+0000","last_peered":"2024-09-15T21:05:02.829531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:02.829531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640664,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3753","reported_seq":3829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.166360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:01.166360+0000","last_peered":"2024-09-15T21:05:01.166360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:01.166360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725388,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3711","reported_seq":5238,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.983337+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:02.983337+0000","last_peered":"2024-09-15T21:05:02.983337+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:02.983337+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355346,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3721","reported_seq":3755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.951311+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:02.951311+0000","last_peered":"2024-09-15T21:05:02.951311+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:02.951311+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472838,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3663","reported_seq":3737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.069301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:01.069301+0000","last_peered":"2024-09-15T21:05:01.069301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:01.069301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29997702,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":29295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3737","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.830201+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:02.830201+0000","last_peered":"2024-09-15T21:05:02.830201+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:02.830201+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577932,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":29876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3690","reported_seq":3767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.179164+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:01.179164+0000","last_peered":"2024-09-15T21:05:01.179164+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:01.179164+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30201100,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3782","reported_seq":3845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.513665+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:02.513665+0000","last_peered":"2024-09-15T21:05:02.513665+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:02.513665+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938380,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":30228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829489152,"data_stored":829489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446028,"kb_used_data":239420,"kb_used_omap":4,"kb_used_meta":206587,"kb_avail":93925812,"statfs":{"total":96636764160,"available":96180031488,"internally_reserved":0,"allocated":245166080,"data_stored":245009130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":211545808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,4,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426868,"kb_used_data":237412,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93944972,"statfs":{"total":96636764160,"available":96199651328,"internally_reserved":0,"allocated":243109888,"data_stored":242952938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":218832,"kb_used_data":92796,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94153008,"statfs":{"total":96636764160,"available":96412680192,"internally_reserved":0,"allocated":95023104,"data_stored":94866154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437872,"kb_used_data":239648,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93933968,"statfs":{"total":96636764160,"available":96188383232,"internally_reserved":0,"allocated":245399552,"data_stored":245242602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":202960590},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245350400,"data_stored":245350400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94924800,"data_stored":94924800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243326976,"data_stored":243326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245886976,"data_stored":245886976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:04.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:04.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3688", 'reported_seq': 3728, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.832618+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:02.832618+0000', 'last_peered': '2024-09-15T21:05:02.832618+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:02.832618+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3688, 'log_dups_size': 0, 'ondisk_log_size': 3688, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30212096, 'num_objects': 3688, 'num_object_clones': 0, 'num_object_copies': 14752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3688, 'num_write_kb': 29504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1412}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:04.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3787", 'reported_seq': 3849, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.530101+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:02.530101+0000', 'last_peered': '2024-09-15T21:05:02.530101+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:02.530101+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3787, 'log_dups_size': 0, 'ondisk_log_size': 3787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30987532, 'num_objects': 3783, 'num_object_clones': 0, 'num_object_copies': 15132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3785, 'num_write_kb': 30276, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3650", 'reported_seq': 3724, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:01.159429+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:01.159429+0000', 'last_peered': '2024-09-15T21:05:01.159429+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:01.159429+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3650, 'log_dups_size': 0, 'ondisk_log_size': 3650, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29874822, 'num_objects': 3647, 'num_object_clones': 0, 'num_object_copies': 14588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3649, 'num_write_kb': 29182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1389}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3788", 'reported_seq': 3851, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.526144+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:02.526144+0000', 'last_peered': '2024-09-15T21:05:02.526144+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:02.526144+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3788, 'log_dups_size': 0, 'ondisk_log_size': 3788, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30995724, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3786, 'num_write_kb': 30277, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3794", 'reported_seq': 3844, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.919116+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:02.919116+0000', 'last_peered': '2024-09-15T21:05:02.919116+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:02.919116+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3794, 'log_dups_size': 0, 'ondisk_log_size': 3794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31053068, 'num_objects': 3791, 'num_object_clones': 0, 'num_object_copies': 15164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3791, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3792, 'num_write_kb': 30333, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1480}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3717", 'reported_seq': 3756, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.939130+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:02.939130+0000', 'last_peered': '2024-09-15T21:05:02.939130+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:02.939130+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3717, 'log_dups_size': 0, 'ondisk_log_size': 3717, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30449664, 'num_objects': 3717, 'num_object_clones': 0, 'num_object_copies': 14868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3717, 'num_write_kb': 29736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1401}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:04.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3795", 'reported_seq': 3859, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.549463+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:02.549463+0000', 'last_peered': '2024-09-15T21:05:02.549463+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:02.549463+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3795, 'log_dups_size': 0, 'ondisk_log_size': 3795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31036684, 'num_objects': 3789, 'num_object_clones': 0, 'num_object_copies': 15156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3793, 'num_write_kb': 30324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1423}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3883", 'reported_seq': 3945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.531990+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:02.531990+0000', 'last_peered': '2024-09-15T21:05:02.531990+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:02.531990+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3883, 'log_dups_size': 0, 'ondisk_log_size': 3883, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31765772, 'num_objects': 3878, 'num_object_clones': 0, 'num_object_copies': 15512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3881, 'num_write_kb': 31036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3749", 'reported_seq': 3803, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.829531+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:02.829531+0000', 'last_peered': '2024-09-15T21:05:02.829531+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:02.829531+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3749, 'log_dups_size': 0, 'ondisk_log_size': 3749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30640664, 'num_objects': 3741, 'num_object_clones': 0, 'num_object_copies': 14964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3746, 'num_write_kb': 29945, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1409}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3753", 'reported_seq': 3829, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:01.166360+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:01.166360+0000', 'last_peered': '2024-09-15T21:05:01.166360+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:01.166360+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3753, 'log_dups_size': 0, 'ondisk_log_size': 3753, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30725388, 'num_objects': 3751, 'num_object_clones': 0, 'num_object_copies': 15004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3751, 'num_write_kb': 30006, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1416}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3711", 'reported_seq': 5238, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.983337+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:02.983337+0000', 'last_peered': '2024-09-15T21:05:02.983337+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:02.983337+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3711, 'log_dups_size': 0, 'ondisk_log_size': 3711, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30355346, 'num_objects': 3706, 'num_object_clones': 0, 'num_object_copies': 14824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1524, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3708, 'num_write_kb': 29652, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 739, 'num_bytes_recovered': 6029312, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2182}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:04.926 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3721", 'reported_seq': 3755, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.951311+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:02.951311+0000', 'last_peered': '2024-09-15T21:05:02.951311+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:02.951311+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3721, 'log_dups_size': 0, 'ondisk_log_size': 3721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30472838, 'num_objects': 3720, 'num_object_clones': 0, 'num_object_copies': 14880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3720, 'num_write_kb': 29759, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1464}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3663", 'reported_seq': 3737, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:01.069301+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:01.069301+0000', 'last_peered': '2024-09-15T21:05:01.069301+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:01.069301+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3663, 'log_dups_size': 0, 'ondisk_log_size': 3663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29997702, 'num_objects': 3662, 'num_object_clones': 0, 'num_object_copies': 14648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3662, 'num_write_kb': 29295, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3737", 'reported_seq': 3788, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.830201+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:02.830201+0000', 'last_peered': '2024-09-15T21:05:02.830201+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:02.830201+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3737, 'log_dups_size': 0, 'ondisk_log_size': 3737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30577932, 'num_objects': 3733, 'num_object_clones': 0, 'num_object_copies': 14932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3733, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3735, 'num_write_kb': 29876, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1454}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3690", 'reported_seq': 3767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:01.179164+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:01.179164+0000', 'last_peered': '2024-09-15T21:05:01.179164+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:01.179164+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3690, 'log_dups_size': 0, 'ondisk_log_size': 3690, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30201100, 'num_objects': 3687, 'num_object_clones': 0, 'num_object_copies': 14748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3688, 'num_write_kb': 29501, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1406}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:04.927 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3782", 'reported_seq': 3845, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:02.513665+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:02.513665+0000', 'last_peered': '2024-09-15T21:05:02.513665+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:02.513665+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3782, 'log_dups_size': 0, 'ondisk_log_size': 3782, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30938380, 'num_objects': 3777, 'num_object_clones': 0, 'num_object_copies': 15108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3780, 'num_write_kb': 30228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:04.927 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:05.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 102 20 60856 60836 4.65905 2.39844 0.0671163 0.0335185 2024-09-15T21:05:05.648 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:05.648 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36347/239416 objects degraded (15.182%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":171,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59854,"data_bytes":490284712,"bytes_used":1566310400,"bytes_avail":384980746240,"bytes_total":386547056640,"degraded_objects":36347,"degraded_total":239416,"degraded_ratio":0.15181525044274402,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":116686,"recovering_keys_per_sec":0,"num_objects_recovered":171,"num_bytes_recovered":1400832,"num_keys_recovered":0,"write_bytes_sec":3191483,"read_op_per_sec":0,"write_op_per_sec":389},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:05.687 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:05.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:06.250 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:06.251 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:06.280 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-15T21:05:05.936180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1529600,"kb_used_data":809276,"kb_used_omap":14,"kb_used_meta":720241,"kb_avail":375957760,"statfs":{"total":386547056640,"available":384980746240,"internally_reserved":0,"allocated":828698624,"data_stored":828070824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":737527346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,11,14,4,10,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27664384,"num_objects":3377,"num_object_clones":0,"num_object_copies":13508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3377,"num_write_kb":27016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004703"},"pg_stats":[{"pgid":"1.f","version":"31'3688","reported_seq":3728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.832618+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:02.832618+0000","last_peered":"2024-09-15T21:05:02.832618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:02.832618+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3787","reported_seq":3849,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.530101+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:02.530101+0000","last_peered":"2024-09-15T21:05:02.530101+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:02.530101+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3787,"log_dups_size":0,"ondisk_log_size":3787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30987532,"num_objects":3783,"num_object_clones":0,"num_object_copies":15132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3785,"num_write_kb":30276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1403}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3650","reported_seq":3724,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.159429+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:01.159429+0000","last_peered":"2024-09-15T21:05:01.159429+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:01.159429+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3650,"log_dups_size":0,"ondisk_log_size":3650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29874822,"num_objects":3647,"num_object_clones":0,"num_object_copies":14588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3649,"num_write_kb":29182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1389}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3788","reported_seq":3851,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.526144+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:02.526144+0000","last_peered":"2024-09-15T21:05:02.526144+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:02.526144+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3788,"log_dups_size":0,"ondisk_log_size":3788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3786,"num_write_kb":30277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3794","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.919116+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:02.919116+0000","last_peered":"2024-09-15T21:05:02.919116+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:02.919116+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053068,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3717","reported_seq":3756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.939130+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:02.939130+0000","last_peered":"2024-09-15T21:05:02.939130+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:02.939130+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3795","reported_seq":3859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.549463+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:02.549463+0000","last_peered":"2024-09-15T21:05:02.549463+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:02.549463+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31036684,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3793,"num_write_kb":30324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1423}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3883","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.531990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:02.531990+0000","last_peered":"2024-09-15T21:05:02.531990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:02.531990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3883,"log_dups_size":0,"ondisk_log_size":3883,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31765772,"num_objects":3878,"num_object_clones":0,"num_object_copies":15512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3881,"num_write_kb":31036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3749","reported_seq":3803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.829531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:02.829531+0000","last_peered":"2024-09-15T21:05:02.829531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:02.829531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640664,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3753","reported_seq":3829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.166360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:01.166360+0000","last_peered":"2024-09-15T21:05:01.166360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:01.166360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3753,"log_dups_size":0,"ondisk_log_size":3753,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30725388,"num_objects":3751,"num_object_clones":0,"num_object_copies":15004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":30006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1416}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3711","reported_seq":5238,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.983337+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:02.983337+0000","last_peered":"2024-09-15T21:05:02.983337+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:02.983337+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355346,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3721","reported_seq":3755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.951311+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:02.951311+0000","last_peered":"2024-09-15T21:05:02.951311+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:02.951311+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472838,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3663","reported_seq":3737,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.069301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:01.069301+0000","last_peered":"2024-09-15T21:05:01.069301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:01.069301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29997702,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":29295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3737","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.830201+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:02.830201+0000","last_peered":"2024-09-15T21:05:02.830201+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:02.830201+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577932,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":29876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3690","reported_seq":3767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:01.179164+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:01.179164+0000","last_peered":"2024-09-15T21:05:01.179164+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:01.179164+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30201100,"num_objects":3687,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1406}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3782","reported_seq":3845,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.513665+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:02.513665+0000","last_peered":"2024-09-15T21:05:02.513665+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:02.513665+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3782,"log_dups_size":0,"ondisk_log_size":3782,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30938380,"num_objects":3777,"num_object_clones":0,"num_object_copies":15108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3780,"num_write_kb":30228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":490284712,"num_objects":59854,"num_object_clones":0,"num_object_copies":239416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59881,"num_write_kb":478930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":829489152,"data_stored":829489152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59908,"ondisk_log_size":59908,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446028,"kb_used_data":239420,"kb_used_omap":4,"kb_used_meta":206587,"kb_avail":93925812,"statfs":{"total":96636764160,"available":96180031488,"internally_reserved":0,"allocated":245166080,"data_stored":245009130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":211545808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,4,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":426868,"kb_used_data":237412,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93944972,"statfs":{"total":96636764160,"available":96199651328,"internally_reserved":0,"allocated":243109888,"data_stored":242952938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051598,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":218832,"kb_used_data":92796,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94153008,"statfs":{"total":96636764160,"available":96412680192,"internally_reserved":0,"allocated":95023104,"data_stored":94866154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":437872,"kb_used_data":239648,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93933968,"statfs":{"total":96636764160,"available":96188383232,"internally_reserved":0,"allocated":245399552,"data_stored":245242602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":202960590},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":245350400,"data_stored":245350400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94924800,"data_stored":94924800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":243326976,"data_stored":243326976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245886976,"data_stored":245886976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:06.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 103 20 61168 61148 4.63747 2.4375 0.108105 0.0336753 2024-09-15T21:05:07.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 104 20 61513 61493 4.6188 2.69531 0.0856293 0.0338163 2024-09-15T21:05:08.641 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 105 20 61857 61837 4.6004 2.6875 0.0305774 0.0339482 2024-09-15T21:05:09.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:09.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 106 20 62180 62160 4.58081 2.52344 0.0810685 0.0340949 2024-09-15T21:05:09.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:09.840 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:09.867 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-15T21:05:07.936997+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":497682088,"num_objects":60757,"num_object_clones":0,"num_object_copies":243028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60784,"num_write_kb":486154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60811,"ondisk_log_size":60811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1597684,"kb_used_data":828788,"kb_used_omap":14,"kb_used_meta":768817,"kb_avail":375889676,"statfs":{"total":386547056640,"available":384911028224,"internally_reserved":0,"allocated":848678912,"data_stored":848051112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":787269170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,22,9,7,12,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35061760,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004974"},"pg_stats":[{"pgid":"1.f","version":"31'3688","reported_seq":3728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.832618+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:02.832618+0000","last_peered":"2024-09-15T21:05:02.832618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:02.832618+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3688,"log_dups_size":0,"ondisk_log_size":3688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":3688,"num_object_clones":0,"num_object_copies":14752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3688,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3894","reported_seq":3956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.626769+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:07.626769+0000","last_peered":"2024-09-15T21:05:07.626769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:07.626769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864076,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3743","reported_seq":3817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.063597+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:06.063597+0000","last_peered":"2024-09-15T21:05:06.063597+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:06.063597+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30636678,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":29926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3898","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.566063+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:07.566063+0000","last_peered":"2024-09-15T21:05:07.566063+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:07.566063+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31896844,"num_objects":3894,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":31157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3794","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.919116+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:02.919116+0000","last_peered":"2024-09-15T21:05:02.919116+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:02.919116+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31053068,"num_objects":3791,"num_object_clones":0,"num_object_copies":15164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":30333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1480}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3717","reported_seq":3756,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.939130+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:02.939130+0000","last_peered":"2024-09-15T21:05:02.939130+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:02.939130+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3717,"log_dups_size":0,"ondisk_log_size":3717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3897","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.621130+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:07.621130+0000","last_peered":"2024-09-15T21:05:07.621130+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:07.621130+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.492435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:07.492435+0000","last_peered":"2024-09-15T21:05:07.492435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:07.492435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3749","reported_seq":3803,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.829531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:02.829531+0000","last_peered":"2024-09-15T21:05:02.829531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:02.829531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3749,"log_dups_size":0,"ondisk_log_size":3749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30640664,"num_objects":3741,"num_object_clones":0,"num_object_copies":14964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3746,"num_write_kb":29945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3850","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.114539+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:06.114539+0000","last_peered":"2024-09-15T21:05:06.114539+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:06.114539+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31520012,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1513}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3711","reported_seq":5238,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.983337+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:02.983337+0000","last_peered":"2024-09-15T21:05:02.983337+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:02.983337+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3711,"log_dups_size":0,"ondisk_log_size":3711,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30355346,"num_objects":3706,"num_object_clones":0,"num_object_copies":14824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3708,"num_write_kb":29652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3721","reported_seq":3755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.951311+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:02.951311+0000","last_peered":"2024-09-15T21:05:02.951311+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:02.951311+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3721,"log_dups_size":0,"ondisk_log_size":3721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30472838,"num_objects":3720,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":29759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1464}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3770","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.174319+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:06.174319+0000","last_peered":"2024-09-15T21:05:06.174319+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:06.174319+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30874246,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3737","reported_seq":3788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:02.830201+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:02.830201+0000","last_peered":"2024-09-15T21:05:02.830201+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:02.830201+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3737,"log_dups_size":0,"ondisk_log_size":3737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30577932,"num_objects":3733,"num_object_clones":0,"num_object_copies":14932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3735,"num_write_kb":29876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1454}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3791","reported_seq":3869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.183692+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:06.183692+0000","last_peered":"2024-09-15T21:05:06.183692+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:06.183692+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31028492,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3874","reported_seq":3937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.642388+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:07.642388+0000","last_peered":"2024-09-15T21:05:07.642388+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:07.642388+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692044,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":497682088,"num_objects":60757,"num_object_clones":0,"num_object_copies":243028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60784,"num_write_kb":486154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":763,"num_bytes_recovered":6029312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":849571840,"data_stored":849571840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60811,"ondisk_log_size":60811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446028,"kb_used_data":239420,"kb_used_omap":4,"kb_used_meta":206587,"kb_avail":93925812,"statfs":{"total":96636764160,"available":96180031488,"internally_reserved":0,"allocated":245166080,"data_stored":245009130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":211545808},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,4,1,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455528,"kb_used_data":243864,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93916312,"statfs":{"total":96636764160,"available":96170303488,"internally_reserved":0,"allocated":249716736,"data_stored":249559786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229664,"kb_used_data":99472,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94142176,"statfs":{"total":96636764160,"available":96401588224,"internally_reserved":0,"allocated":101859328,"data_stored":101702378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466464,"kb_used_data":246032,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93905376,"statfs":{"total":96636764160,"available":96159105024,"internally_reserved":0,"allocated":251936768,"data_stored":251779818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":251961344,"data_stored":251961344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101781504,"data_stored":101781504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249942016,"data_stored":249942016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245886976,"data_stored":245886976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:09.868 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:10.461 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:10.462 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:10.485 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-15T21:05:09.937650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503793320,"num_objects":61503,"num_object_clones":0,"num_object_copies":246012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61530,"num_write_kb":492122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61557,"ondisk_log_size":61557,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1609740,"kb_used_data":835276,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375877620,"statfs":{"total":386547056640,"available":384898682880,"internally_reserved":0,"allocated":855322624,"data_stored":854694824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":792970802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,15,8,12,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27271168,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3329,"num_write_kb":26632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004939"},"pg_stats":[{"pgid":"1.f","version":"31'3789","reported_seq":3829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.981221+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:07.981221+0000","last_peered":"2024-09-15T21:05:07.981221+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:07.981221+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3894","reported_seq":3956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.626769+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:07.626769+0000","last_peered":"2024-09-15T21:05:07.626769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:07.626769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864076,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3743","reported_seq":3817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.063597+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:06.063597+0000","last_peered":"2024-09-15T21:05:06.063597+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:06.063597+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30636678,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":29926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3898","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.566063+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:07.566063+0000","last_peered":"2024-09-15T21:05:07.566063+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:07.566063+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31896844,"num_objects":3894,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":31157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3900","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.975652+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:07.975652+0000","last_peered":"2024-09-15T21:05:07.975652+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:07.975652+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3828","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.998806+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:07.998806+0000","last_peered":"2024-09-15T21:05:07.998806+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:07.998806+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3828,"log_dups_size":0,"ondisk_log_size":3828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31358976,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3828,"num_write_kb":30624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3897","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.621130+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:07.621130+0000","last_peered":"2024-09-15T21:05:07.621130+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:07.621130+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.492435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:07.492435+0000","last_peered":"2024-09-15T21:05:07.492435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:07.492435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3864","reported_seq":3918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.077744+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:08.077744+0000","last_peered":"2024-09-15T21:05:08.077744+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:08.077744+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31582744,"num_objects":3856,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":30865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3850","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.114539+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:06.114539+0000","last_peered":"2024-09-15T21:05:06.114539+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:06.114539+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31520012,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1513}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3813","reported_seq":5451,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.101955+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:08.101955+0000","last_peered":"2024-09-15T21:05:08.101955+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:08.101955+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3813,"log_dups_size":0,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31190930,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":30468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3828","reported_seq":3862,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.036080+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:08.036080+0000","last_peered":"2024-09-15T21:05:08.036080+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:08.036080+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3828,"log_dups_size":0,"ondisk_log_size":3828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31349382,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":30615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3770","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.174319+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:06.174319+0000","last_peered":"2024-09-15T21:05:06.174319+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:06.174319+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30874246,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3841","reported_seq":3892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.966186+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:07.966186+0000","last_peered":"2024-09-15T21:05:07.966186+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:07.966186+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31429900,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3839,"num_write_kb":30708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1558}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3791","reported_seq":3869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.183692+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:06.183692+0000","last_peered":"2024-09-15T21:05:06.183692+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:06.183692+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31028492,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3874","reported_seq":3937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.642388+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:07.642388+0000","last_peered":"2024-09-15T21:05:07.642388+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:07.642388+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692044,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503793320,"num_objects":61503,"num_object_clones":0,"num_object_copies":246012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61530,"num_write_kb":492122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":856305664,"data_stored":856305664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61557,"ondisk_log_size":61557,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458084,"kb_used_data":245908,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93913756,"statfs":{"total":96636764160,"available":96167686144,"internally_reserved":0,"allocated":251809792,"data_stored":251652842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455528,"kb_used_data":243864,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93916312,"statfs":{"total":96636764160,"available":96170303488,"internally_reserved":0,"allocated":249716736,"data_stored":249559786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229664,"kb_used_data":99472,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94142176,"statfs":{"total":96636764160,"available":96401588224,"internally_reserved":0,"allocated":101859328,"data_stored":101702378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466464,"kb_used_data":246032,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93905376,"statfs":{"total":96636764160,"available":96159105024,"internally_reserved":0,"allocated":251936768,"data_stored":251779818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":251961344,"data_stored":251961344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101781504,"data_stored":101781504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249942016,"data_stored":249942016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252620800,"data_stored":252620800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:10.486 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:10.486 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3789", 'reported_seq': 3829, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.981221+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:07.981221+0000', 'last_peered': '2024-09-15T21:05:07.981221+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:07.981221+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3789, 'log_dups_size': 0, 'ondisk_log_size': 3789, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31039488, 'num_objects': 3789, 'num_object_clones': 0, 'num_object_copies': 15156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3789, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3789, 'num_write_kb': 30312, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3894", 'reported_seq': 3956, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.626769+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:07.626769+0000', 'last_peered': '2024-09-15T21:05:07.626769+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:07.626769+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3894, 'log_dups_size': 0, 'ondisk_log_size': 3894, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31864076, 'num_objects': 3890, 'num_object_clones': 0, 'num_object_copies': 15560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3892, 'num_write_kb': 31132, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1510}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3743", 'reported_seq': 3817, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:06.063597+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:06.063597+0000', 'last_peered': '2024-09-15T21:05:06.063597+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:06.063597+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3743, 'log_dups_size': 0, 'ondisk_log_size': 3743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30636678, 'num_objects': 3740, 'num_object_clones': 0, 'num_object_copies': 14960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3740, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3742, 'num_write_kb': 29926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1482}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3898", 'reported_seq': 3962, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.566063+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:07.566063+0000', 'last_peered': '2024-09-15T21:05:07.566063+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:07.566063+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3898, 'log_dups_size': 0, 'ondisk_log_size': 3898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31896844, 'num_objects': 3894, 'num_object_clones': 0, 'num_object_copies': 15576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3896, 'num_write_kb': 31157, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1560}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'3900", 'reported_seq': 3950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.975652+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:07.975652+0000', 'last_peered': '2024-09-15T21:05:07.975652+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:07.975652+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3900, 'log_dups_size': 0, 'ondisk_log_size': 3900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31921420, 'num_objects': 3897, 'num_object_clones': 0, 'num_object_copies': 15588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3898, 'num_write_kb': 31181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1586}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:10.487 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3828", 'reported_seq': 3867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.998806+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:07.998806+0000', 'last_peered': '2024-09-15T21:05:07.998806+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:07.998806+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3828, 'log_dups_size': 0, 'ondisk_log_size': 3828, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31358976, 'num_objects': 3828, 'num_object_clones': 0, 'num_object_copies': 15312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3828, 'num_write_kb': 30624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1512}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3897", 'reported_seq': 3962, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.621130+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:07.621130+0000', 'last_peered': '2024-09-15T21:05:07.621130+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:07.621130+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3897, 'log_dups_size': 0, 'ondisk_log_size': 3897, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31872268, 'num_objects': 3891, 'num_object_clones': 0, 'num_object_copies': 15564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3891, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3895, 'num_write_kb': 31140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1525}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'3977", 'reported_seq': 4040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.492435+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:07.492435+0000', 'last_peered': '2024-09-15T21:05:07.492435+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:07.492435+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3977, 'log_dups_size': 0, 'ondisk_log_size': 3977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32535820, 'num_objects': 3972, 'num_object_clones': 0, 'num_object_copies': 15888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3975, 'num_write_kb': 31788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3864", 'reported_seq': 3918, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:08.077744+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:08.077744+0000', 'last_peered': '2024-09-15T21:05:08.077744+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:08.077744+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3864, 'log_dups_size': 0, 'ondisk_log_size': 3864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31582744, 'num_objects': 3856, 'num_object_clones': 0, 'num_object_copies': 15424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3856, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3861, 'num_write_kb': 30865, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1524}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3850", 'reported_seq': 3926, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:06.114539+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:06.114539+0000', 'last_peered': '2024-09-15T21:05:06.114539+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:06.114539+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3850, 'log_dups_size': 0, 'ondisk_log_size': 3850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31520012, 'num_objects': 3848, 'num_object_clones': 0, 'num_object_copies': 15392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3848, 'num_write_kb': 30782, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:10.488 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3813", 'reported_seq': 5451, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:08.101955+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:08.101955+0000', 'last_peered': '2024-09-15T21:05:08.101955+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:08.101955+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3813, 'log_dups_size': 0, 'ondisk_log_size': 3813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31190930, 'num_objects': 3808, 'num_object_clones': 0, 'num_object_copies': 15232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1468, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3810, 'num_write_kb': 30468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 795, 'num_bytes_recovered': 6488064, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3828", 'reported_seq': 3862, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:08.036080+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:08.036080+0000', 'last_peered': '2024-09-15T21:05:08.036080+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:08.036080+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3828, 'log_dups_size': 0, 'ondisk_log_size': 3828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31349382, 'num_objects': 3827, 'num_object_clones': 0, 'num_object_copies': 15308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3827, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3827, 'num_write_kb': 30615, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1571}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3770", 'reported_seq': 3844, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:06.174319+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:06.174319+0000', 'last_peered': '2024-09-15T21:05:06.174319+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:06.174319+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3770, 'log_dups_size': 0, 'ondisk_log_size': 3770, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30874246, 'num_objects': 3769, 'num_object_clones': 0, 'num_object_copies': 15076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3769, 'num_write_kb': 30151, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3841", 'reported_seq': 3892, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.966186+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:07.966186+0000', 'last_peered': '2024-09-15T21:05:07.966186+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:07.966186+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3841, 'log_dups_size': 0, 'ondisk_log_size': 3841, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31429900, 'num_objects': 3837, 'num_object_clones': 0, 'num_object_copies': 15348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3837, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3839, 'num_write_kb': 30708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1558}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3791", 'reported_seq': 3869, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:06.183692+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:06.183692+0000', 'last_peered': '2024-09-15T21:05:06.183692+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:06.183692+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3791, 'log_dups_size': 0, 'ondisk_log_size': 3791, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31028492, 'num_objects': 3788, 'num_object_clones': 0, 'num_object_copies': 15152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3789, 'num_write_kb': 30309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1507}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:10.489 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3874", 'reported_seq': 3937, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:07.642388+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:07.642388+0000', 'last_peered': '2024-09-15T21:05:07.642388+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:07.642388+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3874, 'log_dups_size': 0, 'ondisk_log_size': 3874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31692044, 'num_objects': 3869, 'num_object_clones': 0, 'num_object_copies': 15476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3872, 'num_write_kb': 30964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1554}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:10.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:10.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 107 20 62477 62457 4.55968 2.32031 0.0437456 0.0342538 2024-09-15T21:05:11.237 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:11.237 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36291/246012 objects degraded (14.752%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":176,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61503,"data_bytes":503793320,"bytes_used":1648373760,"bytes_avail":384898682880,"bytes_total":386547056640,"degraded_objects":36291,"degraded_total":246012,"degraded_ratio":0.1475171942832057,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":77791,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"write_bytes_sec":2271662,"read_op_per_sec":0,"write_op_per_sec":277},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:11.265 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:11.266 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:11.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 108 20 62782 62762 4.53952 2.38281 0.0505971 0.034404 2024-09-15T21:05:11.871 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:11.871 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:11.907 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-15T21:05:09.937650+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":503793320,"num_objects":61503,"num_object_clones":0,"num_object_copies":246012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61530,"num_write_kb":492122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61557,"ondisk_log_size":61557,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1609740,"kb_used_data":835276,"kb_used_omap":14,"kb_used_meta":774385,"kb_avail":375877620,"statfs":{"total":386547056640,"available":384898682880,"internally_reserved":0,"allocated":855322624,"data_stored":854694824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":792970802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,15,8,12,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27271168,"num_objects":3329,"num_object_clones":0,"num_object_copies":13316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3329,"num_write_kb":26632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":933888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004939"},"pg_stats":[{"pgid":"1.f","version":"31'3789","reported_seq":3829,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.981221+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:07.981221+0000","last_peered":"2024-09-15T21:05:07.981221+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:07.981221+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31039488,"num_objects":3789,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1513}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3894","reported_seq":3956,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.626769+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:07.626769+0000","last_peered":"2024-09-15T21:05:07.626769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:07.626769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3894,"log_dups_size":0,"ondisk_log_size":3894,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31864076,"num_objects":3890,"num_object_clones":0,"num_object_copies":15560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3892,"num_write_kb":31132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1510}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3743","reported_seq":3817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.063597+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:06.063597+0000","last_peered":"2024-09-15T21:05:06.063597+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:06.063597+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3743,"log_dups_size":0,"ondisk_log_size":3743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30636678,"num_objects":3740,"num_object_clones":0,"num_object_copies":14960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3742,"num_write_kb":29926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1482}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3898","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.566063+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:07.566063+0000","last_peered":"2024-09-15T21:05:07.566063+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:07.566063+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31896844,"num_objects":3894,"num_object_clones":0,"num_object_copies":15576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3896,"num_write_kb":31157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'3900","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.975652+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:07.975652+0000","last_peered":"2024-09-15T21:05:07.975652+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:07.975652+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31921420,"num_objects":3897,"num_object_clones":0,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3828","reported_seq":3867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.998806+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:07.998806+0000","last_peered":"2024-09-15T21:05:07.998806+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:07.998806+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3828,"log_dups_size":0,"ondisk_log_size":3828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31358976,"num_objects":3828,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3828,"num_write_kb":30624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3897","reported_seq":3962,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.621130+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:07.621130+0000","last_peered":"2024-09-15T21:05:07.621130+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:07.621130+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3897,"log_dups_size":0,"ondisk_log_size":3897,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31872268,"num_objects":3891,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3895,"num_write_kb":31140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1525}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.492435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:07.492435+0000","last_peered":"2024-09-15T21:05:07.492435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:07.492435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3864","reported_seq":3918,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.077744+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:08.077744+0000","last_peered":"2024-09-15T21:05:08.077744+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:08.077744+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31582744,"num_objects":3856,"num_object_clones":0,"num_object_copies":15424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":30865,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3850","reported_seq":3926,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.114539+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:06.114539+0000","last_peered":"2024-09-15T21:05:06.114539+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:06.114539+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31520012,"num_objects":3848,"num_object_clones":0,"num_object_copies":15392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3848,"num_write_kb":30782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1513}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3813","reported_seq":5451,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.101955+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:08.101955+0000","last_peered":"2024-09-15T21:05:08.101955+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:08.101955+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3813,"log_dups_size":0,"ondisk_log_size":3813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31190930,"num_objects":3808,"num_object_clones":0,"num_object_copies":15232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3810,"num_write_kb":30468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3828","reported_seq":3862,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:08.036080+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:08.036080+0000","last_peered":"2024-09-15T21:05:08.036080+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:08.036080+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3828,"log_dups_size":0,"ondisk_log_size":3828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31349382,"num_objects":3827,"num_object_clones":0,"num_object_copies":15308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3827,"num_write_kb":30615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1571}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3770","reported_seq":3844,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.174319+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:06.174319+0000","last_peered":"2024-09-15T21:05:06.174319+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:06.174319+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30874246,"num_objects":3769,"num_object_clones":0,"num_object_copies":15076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":30151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1483}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3841","reported_seq":3892,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.966186+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:07.966186+0000","last_peered":"2024-09-15T21:05:07.966186+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:07.966186+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3841,"log_dups_size":0,"ondisk_log_size":3841,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31429900,"num_objects":3837,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3839,"num_write_kb":30708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1558}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3791","reported_seq":3869,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:06.183692+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:06.183692+0000","last_peered":"2024-09-15T21:05:06.183692+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:06.183692+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31028492,"num_objects":3788,"num_object_clones":0,"num_object_copies":15152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3789,"num_write_kb":30309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1507}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3874","reported_seq":3937,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:07.642388+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:07.642388+0000","last_peered":"2024-09-15T21:05:07.642388+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:07.642388+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3874,"log_dups_size":0,"ondisk_log_size":3874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692044,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3872,"num_write_kb":30964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":503793320,"num_objects":61503,"num_object_clones":0,"num_object_copies":246012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61530,"num_write_kb":492122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":819,"num_bytes_recovered":6488064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":856305664,"data_stored":856305664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61557,"ondisk_log_size":61557,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458084,"kb_used_data":245908,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93913756,"statfs":{"total":96636764160,"available":96167686144,"internally_reserved":0,"allocated":251809792,"data_stored":251652842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,3,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":455528,"kb_used_data":243864,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93916312,"statfs":{"total":96636764160,"available":96170303488,"internally_reserved":0,"allocated":249716736,"data_stored":249559786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":229664,"kb_used_data":99472,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94142176,"statfs":{"total":96636764160,"available":96401588224,"internally_reserved":0,"allocated":101859328,"data_stored":101702378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466464,"kb_used_data":246032,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93905376,"statfs":{"total":96636764160,"available":96159105024,"internally_reserved":0,"allocated":251936768,"data_stored":251779818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,0,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":251961344,"data_stored":251961344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":101781504,"data_stored":101781504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":249942016,"data_stored":249942016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252620800,"data_stored":252620800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:12.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 109 20 63112 63092 4.52153 2.57812 0.0555652 0.0345421 2024-09-15T21:05:13.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 110 20 63456 63436 4.50485 2.6875 0.0368774 0.0346684 2024-09-15T21:05:14.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 111 20 63801 63781 4.48855 2.69531 0.0649525 0.0347969 2024-09-15T21:05:14.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:15.499 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:15.500 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:15.529 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-15T21:05:13.939560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1627500,"kb_used_data":861356,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375859860,"statfs":{"total":386547056640,"available":384880496640,"internally_reserved":0,"allocated":882028544,"data_stored":881400744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":784451122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,10,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":36839424,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4497,"num_write_kb":35976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":1458176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005144"},"pg_stats":[{"pgid":"1.f","version":"31'3898","reported_seq":3938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.913707+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:12.913707+0000","last_peered":"2024-09-15T21:05:12.913707+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:12.913707+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31932416,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3983","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.657453+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:12.657453+0000","last_peered":"2024-09-15T21:05:12.657453+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:12.657453+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593164,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3855","reported_seq":3929,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.236077+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:11.236077+0000","last_peered":"2024-09-15T21:05:11.236077+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:11.236077+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3854,"num_write_kb":30822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1594}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3989","reported_seq":4053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.467240+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:12.467240+0000","last_peered":"2024-09-15T21:05:12.467240+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:12.467240+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32642316,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4005","reported_seq":4054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.092520+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:13.092520+0000","last_peered":"2024-09-15T21:05:13.092520+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:13.092520+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32781580,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":32021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3932","reported_seq":3971,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.122182+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:13.122182+0000","last_peered":"2024-09-15T21:05:13.122182+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:13.122182+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32210944,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3992","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.574830+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:12.574830+0000","last_peered":"2024-09-15T21:05:12.574830+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:12.574830+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32650508,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1620}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4101","reported_seq":4165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.634894+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:12.634894+0000","last_peered":"2024-09-15T21:05:12.634894+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:12.634894+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551628,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3949","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.096672+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:13.096672+0000","last_peered":"2024-09-15T21:05:13.096672+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:13.096672+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32279064,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":31545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3933","reported_seq":4009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.119225+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:11.119225+0000","last_peered":"2024-09-15T21:05:11.119225+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:11.119225+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32199948,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3905","reported_seq":5672,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.148974+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:13.148974+0000","last_peered":"2024-09-15T21:05:13.148974+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:13.148974+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31944594,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3936","reported_seq":3970,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.107541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:13.107541+0000","last_peered":"2024-09-15T21:05:13.107541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:13.107541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32234118,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3935,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1679}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3875","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.171760+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:11.171760+0000","last_peered":"2024-09-15T21:05:11.171760+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:11.171760+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31734406,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3874,"num_write_kb":30991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3941","reported_seq":3992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.002573+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:13.002573+0000","last_peered":"2024-09-15T21:05:13.002573+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:13.002573+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249100,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3867","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.137405+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:11.137405+0000","last_peered":"2024-09-15T21:05:11.137405+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:11.137405+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651084,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.641597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:12.641597+0000","last_peered":"2024-09-15T21:05:12.641597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:12.641597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883113984,"data_stored":883113984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468544,"kb_used_data":252208,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93903296,"statfs":{"total":96636764160,"available":96156975104,"internally_reserved":0,"allocated":258260992,"data_stored":258104042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466316,"kb_used_data":250492,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93905524,"statfs":{"total":96636764160,"available":96159256576,"internally_reserved":0,"allocated":256503808,"data_stored":256346858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228108,"kb_used_data":106236,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94143732,"statfs":{"total":96636764160,"available":96403181568,"internally_reserved":0,"allocated":108785664,"data_stored":108628714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464532,"kb_used_data":252420,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93907308,"statfs":{"total":96636764160,"available":96161083392,"internally_reserved":0,"allocated":258478080,"data_stored":258321130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258527232,"data_stored":258527232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108695552,"data_stored":108695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256675840,"data_stored":256675840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259215360,"data_stored":259215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:15.530 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:15.639 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 112 20 64113 64093 4.47024 2.4375 0.0629022 0.0349349 2024-09-15T21:05:16.084 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:16.085 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:16.120 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-15T21:05:15.940360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1627500,"kb_used_data":861356,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375859860,"statfs":{"total":386547056640,"available":384880496640,"internally_reserved":0,"allocated":882028544,"data_stored":881400744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":784451122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,10,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26460160,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005034"},"pg_stats":[{"pgid":"1.f","version":"31'3898","reported_seq":3938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.913707+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:12.913707+0000","last_peered":"2024-09-15T21:05:12.913707+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:12.913707+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31932416,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3983","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.657453+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:12.657453+0000","last_peered":"2024-09-15T21:05:12.657453+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:12.657453+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593164,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3855","reported_seq":3929,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.236077+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:11.236077+0000","last_peered":"2024-09-15T21:05:11.236077+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:11.236077+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3854,"num_write_kb":30822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1594}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3989","reported_seq":4053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.467240+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:12.467240+0000","last_peered":"2024-09-15T21:05:12.467240+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:12.467240+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32642316,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4005","reported_seq":4054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.092520+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:13.092520+0000","last_peered":"2024-09-15T21:05:13.092520+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:13.092520+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32781580,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":32021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3932","reported_seq":3971,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.122182+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:13.122182+0000","last_peered":"2024-09-15T21:05:13.122182+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:13.122182+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32210944,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3992","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.574830+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:12.574830+0000","last_peered":"2024-09-15T21:05:12.574830+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:12.574830+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32650508,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1620}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4101","reported_seq":4165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.634894+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:12.634894+0000","last_peered":"2024-09-15T21:05:12.634894+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:12.634894+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551628,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3949","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.096672+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:13.096672+0000","last_peered":"2024-09-15T21:05:13.096672+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:13.096672+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32279064,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":31545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3933","reported_seq":4009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.119225+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:11.119225+0000","last_peered":"2024-09-15T21:05:11.119225+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:11.119225+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32199948,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3905","reported_seq":5672,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.148974+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:13.148974+0000","last_peered":"2024-09-15T21:05:13.148974+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:13.148974+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31944594,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3936","reported_seq":3970,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.107541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:13.107541+0000","last_peered":"2024-09-15T21:05:13.107541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:13.107541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32234118,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3935,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1679}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3875","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.171760+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:11.171760+0000","last_peered":"2024-09-15T21:05:11.171760+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:11.171760+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31734406,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3874,"num_write_kb":30991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3941","reported_seq":3992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.002573+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:13.002573+0000","last_peered":"2024-09-15T21:05:13.002573+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:13.002573+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249100,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3867","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.137405+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:11.137405+0000","last_peered":"2024-09-15T21:05:11.137405+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:11.137405+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651084,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.641597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:12.641597+0000","last_peered":"2024-09-15T21:05:12.641597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:12.641597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883113984,"data_stored":883113984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468544,"kb_used_data":252208,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93903296,"statfs":{"total":96636764160,"available":96156975104,"internally_reserved":0,"allocated":258260992,"data_stored":258104042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466316,"kb_used_data":250492,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93905524,"statfs":{"total":96636764160,"available":96159256576,"internally_reserved":0,"allocated":256503808,"data_stored":256346858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228108,"kb_used_data":106236,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94143732,"statfs":{"total":96636764160,"available":96403181568,"internally_reserved":0,"allocated":108785664,"data_stored":108628714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464532,"kb_used_data":252420,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93907308,"statfs":{"total":96636764160,"available":96161083392,"internally_reserved":0,"allocated":258478080,"data_stored":258321130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258527232,"data_stored":258527232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108695552,"data_stored":108695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256675840,"data_stored":256675840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259215360,"data_stored":259215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'3898", 'reported_seq': 3938, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.913707+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:12.913707+0000', 'last_peered': '2024-09-15T21:05:12.913707+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:12.913707+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3898, 'log_dups_size': 0, 'ondisk_log_size': 3898, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31932416, 'num_objects': 3898, 'num_object_clones': 0, 'num_object_copies': 15592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3898, 'num_write_kb': 31184, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1622}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'3983", 'reported_seq': 4044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.657453+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:12.657453+0000', 'last_peered': '2024-09-15T21:05:12.657453+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:12.657453+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3983, 'log_dups_size': 0, 'ondisk_log_size': 3983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32593164, 'num_objects': 3979, 'num_object_clones': 0, 'num_object_copies': 15916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3979, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3981, 'num_write_kb': 31844, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1599}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:16.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3855", 'reported_seq': 3929, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:11.236077+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:11.236077+0000', 'last_peered': '2024-09-15T21:05:11.236077+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:11.236077+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3855, 'log_dups_size': 0, 'ondisk_log_size': 3855, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31554182, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3854, 'num_write_kb': 30822, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1594}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'3989", 'reported_seq': 4053, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.467240+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:12.467240+0000', 'last_peered': '2024-09-15T21:05:12.467240+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:12.467240+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3989, 'log_dups_size': 0, 'ondisk_log_size': 3989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32642316, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 15940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3987, 'num_write_kb': 31885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1651}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4005", 'reported_seq': 4054, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.092520+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:13.092520+0000', 'last_peered': '2024-09-15T21:05:13.092520+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:13.092520+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4005, 'log_dups_size': 0, 'ondisk_log_size': 4005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32781580, 'num_objects': 4002, 'num_object_clones': 0, 'num_object_copies': 16008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4003, 'num_write_kb': 32021, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'3932", 'reported_seq': 3971, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.122182+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:13.122182+0000', 'last_peered': '2024-09-15T21:05:13.122182+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:13.122182+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3932, 'log_dups_size': 0, 'ondisk_log_size': 3932, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32210944, 'num_objects': 3932, 'num_object_clones': 0, 'num_object_copies': 15728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3932, 'num_write_kb': 31456, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'3992", 'reported_seq': 4057, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.574830+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:12.574830+0000', 'last_peered': '2024-09-15T21:05:12.574830+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:12.574830+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3992, 'log_dups_size': 0, 'ondisk_log_size': 3992, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32650508, 'num_objects': 3986, 'num_object_clones': 0, 'num_object_copies': 15944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3986, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3990, 'num_write_kb': 31900, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1620}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:16.122 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4101", 'reported_seq': 4165, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.634894+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:12.634894+0000', 'last_peered': '2024-09-15T21:05:12.634894+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:12.634894+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4101, 'log_dups_size': 0, 'ondisk_log_size': 4101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33551628, 'num_objects': 4096, 'num_object_clones': 0, 'num_object_copies': 16384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4099, 'num_write_kb': 32780, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'3949", 'reported_seq': 4003, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.096672+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:13.096672+0000', 'last_peered': '2024-09-15T21:05:13.096672+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:13.096672+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3949, 'log_dups_size': 0, 'ondisk_log_size': 3949, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32279064, 'num_objects': 3941, 'num_object_clones': 0, 'num_object_copies': 15764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3946, 'num_write_kb': 31545, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'3933", 'reported_seq': 4009, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:11.119225+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:11.119225+0000', 'last_peered': '2024-09-15T21:05:11.119225+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:11.119225+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3933, 'log_dups_size': 0, 'ondisk_log_size': 3933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32199948, 'num_objects': 3931, 'num_object_clones': 0, 'num_object_copies': 15724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3931, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3931, 'num_write_kb': 31446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1596}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'3905", 'reported_seq': 5672, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.148974+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:13.148974+0000', 'last_peered': '2024-09-15T21:05:13.148974+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:13.148974+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3905, 'log_dups_size': 0, 'ondisk_log_size': 3905, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31944594, 'num_objects': 3900, 'num_object_clones': 0, 'num_object_copies': 15600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1404, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3902, 'num_write_kb': 31204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 859, 'num_bytes_recovered': 7012352, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:16.123 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'3936", 'reported_seq': 3970, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.107541+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:13.107541+0000', 'last_peered': '2024-09-15T21:05:13.107541+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:13.107541+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3936, 'log_dups_size': 0, 'ondisk_log_size': 3936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32234118, 'num_objects': 3935, 'num_object_clones': 0, 'num_object_copies': 15740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3935, 'num_write_kb': 31479, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3875", 'reported_seq': 3950, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:11.171760+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:11.171760+0000', 'last_peered': '2024-09-15T21:05:11.171760+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:11.171760+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3875, 'log_dups_size': 0, 'ondisk_log_size': 3875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31734406, 'num_objects': 3874, 'num_object_clones': 0, 'num_object_copies': 15496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3874, 'num_write_kb': 30991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'3941", 'reported_seq': 3992, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:13.002573+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:13.002573+0000', 'last_peered': '2024-09-15T21:05:13.002573+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:13.002573+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3941, 'log_dups_size': 0, 'ondisk_log_size': 3941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32249100, 'num_objects': 3937, 'num_object_clones': 0, 'num_object_copies': 15748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3937, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3939, 'num_write_kb': 31508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1658}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3867", 'reported_seq': 3945, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:11.137405+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:11.137405+0000', 'last_peered': '2024-09-15T21:05:11.137405+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:11.137405+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3867, 'log_dups_size': 0, 'ondisk_log_size': 3867, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31651084, 'num_objects': 3864, 'num_object_clones': 0, 'num_object_copies': 15456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3864, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3865, 'num_write_kb': 30917, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1583}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:16.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'3977", 'reported_seq': 4040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:12.641597+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:12.641597+0000', 'last_peered': '2024-09-15T21:05:12.641597+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:12.641597+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3977, 'log_dups_size': 0, 'ondisk_log_size': 3977, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32535820, 'num_objects': 3972, 'num_object_clones': 0, 'num_object_copies': 15888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3975, 'num_write_kb': 31788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:16.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:16.640 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 113 20 64422 64402 4.45204 2.41406 0.0912399 0.0350801 2024-09-15T21:05:16.871 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:16.871 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36227/252336 objects degraded (14.357%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":182,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63084,"data_bytes":516744872,"bytes_used":1666560000,"bytes_avail":384880496640,"bytes_total":386547056640,"degraded_objects":36227,"degraded_total":252336,"degraded_ratio":0.14356651448861835,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":81885,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":2204088,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:16.895 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:16.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:17.493 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:17.494 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:17.525 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-15T21:05:15.940360+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1627500,"kb_used_data":861356,"kb_used_omap":14,"kb_used_meta":766065,"kb_avail":375859860,"statfs":{"total":386547056640,"available":384880496640,"internally_reserved":0,"allocated":882028544,"data_stored":881400744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":784451122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5,10,13,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26460160,"num_objects":3230,"num_object_clones":0,"num_object_copies":12920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3230,"num_write_kb":25840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005034"},"pg_stats":[{"pgid":"1.f","version":"31'3898","reported_seq":3938,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.913707+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:12.913707+0000","last_peered":"2024-09-15T21:05:12.913707+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:12.913707+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31932416,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":31184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1622}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'3983","reported_seq":4044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.657453+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:12.657453+0000","last_peered":"2024-09-15T21:05:12.657453+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:12.657453+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32593164,"num_objects":3979,"num_object_clones":0,"num_object_copies":15916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3981,"num_write_kb":31844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1599}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3855","reported_seq":3929,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.236077+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:11.236077+0000","last_peered":"2024-09-15T21:05:11.236077+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:11.236077+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3854,"num_write_kb":30822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1594}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'3989","reported_seq":4053,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.467240+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:12.467240+0000","last_peered":"2024-09-15T21:05:12.467240+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:12.467240+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32642316,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3987,"num_write_kb":31885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1651}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4005","reported_seq":4054,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.092520+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:13.092520+0000","last_peered":"2024-09-15T21:05:13.092520+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:13.092520+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32781580,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":32021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'3932","reported_seq":3971,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.122182+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:13.122182+0000","last_peered":"2024-09-15T21:05:13.122182+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:13.122182+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32210944,"num_objects":3932,"num_object_clones":0,"num_object_copies":15728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3932,"num_write_kb":31456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'3992","reported_seq":4057,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.574830+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:12.574830+0000","last_peered":"2024-09-15T21:05:12.574830+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:12.574830+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32650508,"num_objects":3986,"num_object_clones":0,"num_object_copies":15944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3990,"num_write_kb":31900,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1620}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4101","reported_seq":4165,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.634894+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:12.634894+0000","last_peered":"2024-09-15T21:05:12.634894+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:12.634894+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4101,"log_dups_size":0,"ondisk_log_size":4101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33551628,"num_objects":4096,"num_object_clones":0,"num_object_copies":16384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4099,"num_write_kb":32780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1622}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'3949","reported_seq":4003,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.096672+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:13.096672+0000","last_peered":"2024-09-15T21:05:13.096672+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:13.096672+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32279064,"num_objects":3941,"num_object_clones":0,"num_object_copies":15764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3946,"num_write_kb":31545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'3933","reported_seq":4009,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.119225+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:11.119225+0000","last_peered":"2024-09-15T21:05:11.119225+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:11.119225+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3933,"log_dups_size":0,"ondisk_log_size":3933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32199948,"num_objects":3931,"num_object_clones":0,"num_object_copies":15724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3931,"num_write_kb":31446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1596}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'3905","reported_seq":5672,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.148974+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:13.148974+0000","last_peered":"2024-09-15T21:05:13.148974+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:13.148974+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31944594,"num_objects":3900,"num_object_clones":0,"num_object_copies":15600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3902,"num_write_kb":31204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":859,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'3936","reported_seq":3970,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.107541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:13.107541+0000","last_peered":"2024-09-15T21:05:13.107541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:13.107541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32234118,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3935,"num_write_kb":31479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1679}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3875","reported_seq":3950,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.171760+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:11.171760+0000","last_peered":"2024-09-15T21:05:11.171760+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:11.171760+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3875,"log_dups_size":0,"ondisk_log_size":3875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31734406,"num_objects":3874,"num_object_clones":0,"num_object_copies":15496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3874,"num_write_kb":30991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'3941","reported_seq":3992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:13.002573+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:13.002573+0000","last_peered":"2024-09-15T21:05:13.002573+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:13.002573+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3941,"log_dups_size":0,"ondisk_log_size":3941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32249100,"num_objects":3937,"num_object_clones":0,"num_object_copies":15748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3939,"num_write_kb":31508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1658}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3867","reported_seq":3945,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:11.137405+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:11.137405+0000","last_peered":"2024-09-15T21:05:11.137405+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:11.137405+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3867,"log_dups_size":0,"ondisk_log_size":3867,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651084,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":30917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1583}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'3977","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:12.641597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:12.641597+0000","last_peered":"2024-09-15T21:05:12.641597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:12.641597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3977,"log_dups_size":0,"ondisk_log_size":3977,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32535820,"num_objects":3972,"num_object_clones":0,"num_object_copies":15888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3975,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":516744872,"num_objects":63084,"num_object_clones":0,"num_object_copies":252336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63111,"num_write_kb":504770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":883,"num_bytes_recovered":7012352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883113984,"data_stored":883113984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63138,"ondisk_log_size":63138,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468544,"kb_used_data":252208,"kb_used_omap":4,"kb_used_meta":216315,"kb_avail":93903296,"statfs":{"total":96636764160,"available":96156975104,"internally_reserved":0,"allocated":258260992,"data_stored":258104042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":221507280},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,3,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466316,"kb_used_data":250492,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93905524,"statfs":{"total":96636764160,"available":96159256576,"internally_reserved":0,"allocated":256503808,"data_stored":256346858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051600,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":228108,"kb_used_data":106236,"kb_used_omap":4,"kb_used_meta":121851,"kb_avail":94143732,"statfs":{"total":96636764160,"available":96403181568,"internally_reserved":0,"allocated":108785664,"data_stored":108628714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":124776143},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":464532,"kb_used_data":252420,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93907308,"statfs":{"total":96636764160,"available":96161083392,"internally_reserved":0,"allocated":258478080,"data_stored":258321130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258527232,"data_stored":258527232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108695552,"data_stored":108695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":256675840,"data_stored":256675840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259215360,"data_stored":259215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:17.640 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 114 20 64732 64712 4.43423 2.42188 0.0724733 0.0352206 2024-09-15T21:05:18.641 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 115 20 65071 65051 4.4187 2.64844 0.0341158 0.0353485 2024-09-15T21:05:19.640 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 116 20 65402 65382 4.4029 2.58594 0.0634733 0.0354698 2024-09-15T21:05:20.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:20.640 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 117 20 65738 65718 4.3877 2.625 0.0375958 0.0355932 2024-09-15T21:05:21.142 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:21.142 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:21.167 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-15T21:05:19.942361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530237096,"num_objects":64731,"num_object_clones":0,"num_object_copies":258924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64758,"num_write_kb":517946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64785,"ondisk_log_size":64785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1670512,"kb_used_data":887724,"kb_used_omap":14,"kb_used_meta":782705,"kb_avail":375816848,"statfs":{"total":386547056640,"available":384836452352,"internally_reserved":0,"allocated":909029376,"data_stored":908401576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":801490482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,7,11,13,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":31792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005363"},"pg_stats":[{"pgid":"1.f","version":"31'4012","reported_seq":4052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.151204+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:18.151204+0000","last_peered":"2024-09-15T21:05:18.151204+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:18.151204+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32866304,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4078","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.664510+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:17.664510+0000","last_peered":"2024-09-15T21:05:17.664510+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:17.664510+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3951","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.206251+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:16.206251+0000","last_peered":"2024-09-15T21:05:16.206251+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:16.206251+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32340614,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3950,"num_write_kb":31590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1690}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4089","reported_seq":4153,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.607039+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:17.607039+0000","last_peered":"2024-09-15T21:05:17.607039+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:17.607039+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461516,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4109","reported_seq":4159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.060164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:18.060164+0000","last_peered":"2024-09-15T21:05:18.060164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:18.060164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33633548,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4038","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.077797+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:18.077797+0000","last_peered":"2024-09-15T21:05:18.077797+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:18.077797+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4109","reported_seq":4173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.673368+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:17.673368+0000","last_peered":"2024-09-15T21:05:17.673368+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:17.673368+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608972,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.570472+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:17.570472+0000","last_peered":"2024-09-15T21:05:17.570472+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:17.570472+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256140,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4052","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.142694+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:18.142694+0000","last_peered":"2024-09-15T21:05:18.142694+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:18.142694+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33122840,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":32369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4049","reported_seq":4125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.212560+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:16.212560+0000","last_peered":"2024-09-15T21:05:16.212560+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:16.212560+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4049,"log_dups_size":0,"ondisk_log_size":4049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150220,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4047,"num_write_kb":32374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4019","reported_seq":5900,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.170626+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:18.170626+0000","last_peered":"2024-09-15T21:05:18.170626+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:18.170626+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32878482,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4016,"num_write_kb":32116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":916,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4006","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.016627+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:18.016627+0000","last_peered":"2024-09-15T21:05:18.016627+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:18.016627+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32807558,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":32039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3983","reported_seq":4058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.121365+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:16.121365+0000","last_peered":"2024-09-15T21:05:16.121365+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:16.121365+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32619142,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1696}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4048","reported_seq":4098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.192746+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:18.192746+0000","last_peered":"2024-09-15T21:05:18.192746+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:18.192746+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33125644,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":32364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3988","reported_seq":4066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.161168+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:16.161168+0000","last_peered":"2024-09-15T21:05:16.161168+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:16.161168+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32642316,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":31885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4067","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.653601+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:17.653601+0000","last_peered":"2024-09-15T21:05:17.653601+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:17.653601+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":32508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530237096,"num_objects":64731,"num_object_clones":0,"num_object_copies":258924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64758,"num_write_kb":517946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":910290944,"data_stored":910290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64785,"ondisk_log_size":64785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479260,"kb_used_data":258764,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93892580,"statfs":{"total":96636764160,"available":96146001920,"internally_reserved":0,"allocated":264974336,"data_stored":264817386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477028,"kb_used_data":257044,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93894812,"statfs":{"total":96636764160,"available":96148287488,"internally_reserved":0,"allocated":263213056,"data_stored":263056106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239000,"kb_used_data":112968,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94132840,"statfs":{"total":96636764160,"available":96392028160,"internally_reserved":0,"allocated":115679232,"data_stored":115522282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475224,"kb_used_data":258948,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93896616,"statfs":{"total":96636764160,"available":96150134784,"internally_reserved":0,"allocated":265162752,"data_stored":265005802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":221441742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265256960,"data_stored":265256960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115716096,"data_stored":115716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263397376,"data_stored":263397376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265920512,"data_stored":265920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:21.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:21.644 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 118 20 66037 66017 4.37031 2.33594 0.064725 0.0357386 2024-09-15T21:05:21.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:21.802 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:21.821 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-15T21:05:19.942361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530237096,"num_objects":64731,"num_object_clones":0,"num_object_copies":258924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64758,"num_write_kb":517946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64785,"ondisk_log_size":64785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1670512,"kb_used_data":887724,"kb_used_omap":14,"kb_used_meta":782705,"kb_avail":375816848,"statfs":{"total":386547056640,"available":384836452352,"internally_reserved":0,"allocated":909029376,"data_stored":908401576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":801490482},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,7,11,13,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3974,"num_write_kb":31792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005363"},"pg_stats":[{"pgid":"1.f","version":"31'4012","reported_seq":4052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.151204+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:18.151204+0000","last_peered":"2024-09-15T21:05:18.151204+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:18.151204+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32866304,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4078","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.664510+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:17.664510+0000","last_peered":"2024-09-15T21:05:17.664510+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:17.664510+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'3951","reported_seq":4025,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.206251+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:16.206251+0000","last_peered":"2024-09-15T21:05:16.206251+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:16.206251+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3951,"log_dups_size":0,"ondisk_log_size":3951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32340614,"num_objects":3948,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3950,"num_write_kb":31590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1690}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4089","reported_seq":4153,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.607039+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:17.607039+0000","last_peered":"2024-09-15T21:05:17.607039+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:17.607039+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461516,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4109","reported_seq":4159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.060164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:18.060164+0000","last_peered":"2024-09-15T21:05:18.060164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:18.060164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33633548,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4038","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.077797+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:18.077797+0000","last_peered":"2024-09-15T21:05:18.077797+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:18.077797+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4109","reported_seq":4173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.673368+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:17.673368+0000","last_peered":"2024-09-15T21:05:17.673368+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:17.673368+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608972,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.570472+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:17.570472+0000","last_peered":"2024-09-15T21:05:17.570472+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:17.570472+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256140,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4052","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.142694+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:18.142694+0000","last_peered":"2024-09-15T21:05:18.142694+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:18.142694+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33122840,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":32369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4049","reported_seq":4125,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.212560+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:16.212560+0000","last_peered":"2024-09-15T21:05:16.212560+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:16.212560+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4049,"log_dups_size":0,"ondisk_log_size":4049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33150220,"num_objects":4047,"num_object_clones":0,"num_object_copies":16188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4047,"num_write_kb":32374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1712}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4019","reported_seq":5900,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.170626+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:18.170626+0000","last_peered":"2024-09-15T21:05:18.170626+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:18.170626+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32878482,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4016,"num_write_kb":32116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":916,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4006","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.016627+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:18.016627+0000","last_peered":"2024-09-15T21:05:18.016627+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:18.016627+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32807558,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":32039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'3983","reported_seq":4058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.121365+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:16.121365+0000","last_peered":"2024-09-15T21:05:16.121365+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:16.121365+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32619142,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1696}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4048","reported_seq":4098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.192746+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:18.192746+0000","last_peered":"2024-09-15T21:05:18.192746+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:18.192746+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33125644,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":32364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'3988","reported_seq":4066,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:16.161168+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:16.161168+0000","last_peered":"2024-09-15T21:05:16.161168+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:16.161168+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":3988,"log_dups_size":0,"ondisk_log_size":3988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32642316,"num_objects":3985,"num_object_clones":0,"num_object_copies":15940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":31885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4067","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.653601+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:17.653601+0000","last_peered":"2024-09-15T21:05:17.653601+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:17.653601+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":32508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":530237096,"num_objects":64731,"num_object_clones":0,"num_object_copies":258924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64758,"num_write_kb":517946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":910290944,"data_stored":910290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64785,"ondisk_log_size":64785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479260,"kb_used_data":258764,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93892580,"statfs":{"total":96636764160,"available":96146001920,"internally_reserved":0,"allocated":264974336,"data_stored":264817386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":477028,"kb_used_data":257044,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93894812,"statfs":{"total":96636764160,"available":96148287488,"internally_reserved":0,"allocated":263213056,"data_stored":263056106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":239000,"kb_used_data":112968,"kb_used_omap":4,"kb_used_meta":126011,"kb_avail":94132840,"statfs":{"total":96636764160,"available":96392028160,"internally_reserved":0,"allocated":115679232,"data_stored":115522282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":129035983},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475224,"kb_used_data":258948,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93896616,"statfs":{"total":96636764160,"available":96150134784,"internally_reserved":0,"allocated":265162752,"data_stored":265005802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":221441742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265256960,"data_stored":265256960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115716096,"data_stored":115716096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":263397376,"data_stored":263397376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265920512,"data_stored":265920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4012", 'reported_seq': 4052, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.151204+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:18.151204+0000', 'last_peered': '2024-09-15T21:05:18.151204+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:18.151204+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4012, 'log_dups_size': 0, 'ondisk_log_size': 4012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32866304, 'num_objects': 4012, 'num_object_clones': 0, 'num_object_copies': 16048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4012, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4012, 'num_write_kb': 32096, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1736}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4078", 'reported_seq': 4140, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:17.664510+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:17.664510+0000', 'last_peered': '2024-09-15T21:05:17.664510+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:17.664510+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4078, 'log_dups_size': 0, 'ondisk_log_size': 4078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33371404, 'num_objects': 4074, 'num_object_clones': 0, 'num_object_copies': 16296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4074, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4076, 'num_write_kb': 32604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1694}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'3951", 'reported_seq': 4025, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:16.206251+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:16.206251+0000', 'last_peered': '2024-09-15T21:05:16.206251+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:16.206251+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3951, 'log_dups_size': 0, 'ondisk_log_size': 3951, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32340614, 'num_objects': 3948, 'num_object_clones': 0, 'num_object_copies': 15792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3950, 'num_write_kb': 31590, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1690}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4089", 'reported_seq': 4153, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:17.607039+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:17.607039+0000', 'last_peered': '2024-09-15T21:05:17.607039+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:17.607039+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4089, 'log_dups_size': 0, 'ondisk_log_size': 4089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33461516, 'num_objects': 4085, 'num_object_clones': 0, 'num_object_copies': 16340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4085, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4087, 'num_write_kb': 32685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:21.822 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4109", 'reported_seq': 4159, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.060164+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:18.060164+0000', 'last_peered': '2024-09-15T21:05:18.060164+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:18.060164+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4109, 'log_dups_size': 0, 'ondisk_log_size': 4109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33633548, 'num_objects': 4106, 'num_object_clones': 0, 'num_object_copies': 16424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4107, 'num_write_kb': 32853, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1795}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4038", 'reported_seq': 4077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.077797+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:18.077797+0000', 'last_peered': '2024-09-15T21:05:18.077797+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:18.077797+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4038, 'log_dups_size': 0, 'ondisk_log_size': 4038, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33079296, 'num_objects': 4038, 'num_object_clones': 0, 'num_object_copies': 16152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4038, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4038, 'num_write_kb': 32304, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1722}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4109", 'reported_seq': 4173, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:17.673368+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:17.673368+0000', 'last_peered': '2024-09-15T21:05:17.673368+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:17.673368+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4109, 'log_dups_size': 0, 'ondisk_log_size': 4109, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33608972, 'num_objects': 4103, 'num_object_clones': 0, 'num_object_copies': 16412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4107, 'num_write_kb': 32836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1737}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4187", 'reported_seq': 4251, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:17.570472+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:17.570472+0000', 'last_peered': '2024-09-15T21:05:17.570472+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:17.570472+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4187, 'log_dups_size': 0, 'ondisk_log_size': 4187, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34256140, 'num_objects': 4182, 'num_object_clones': 0, 'num_object_copies': 16728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4185, 'num_write_kb': 33468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1708}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4052", 'reported_seq': 4104, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.142694+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:18.142694+0000', 'last_peered': '2024-09-15T21:05:18.142694+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:18.142694+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4052, 'log_dups_size': 0, 'ondisk_log_size': 4052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33122840, 'num_objects': 4044, 'num_object_clones': 0, 'num_object_copies': 16176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4049, 'num_write_kb': 32369, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:21.823 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4049", 'reported_seq': 4125, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:16.212560+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:16.212560+0000', 'last_peered': '2024-09-15T21:05:16.212560+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:16.212560+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4049, 'log_dups_size': 0, 'ondisk_log_size': 4049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33150220, 'num_objects': 4047, 'num_object_clones': 0, 'num_object_copies': 16188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4047, 'num_write_kb': 32374, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1712}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4019", 'reported_seq': 5900, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.170626+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:18.170626+0000', 'last_peered': '2024-09-15T21:05:18.170626+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:18.170626+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4019, 'log_dups_size': 0, 'ondisk_log_size': 4019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32878482, 'num_objects': 4014, 'num_object_clones': 0, 'num_object_copies': 16056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1347, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4014, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4016, 'num_write_kb': 32116, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 916, 'num_bytes_recovered': 7479296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2667}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4006", 'reported_seq': 4040, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.016627+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:18.016627+0000', 'last_peered': '2024-09-15T21:05:18.016627+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:18.016627+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4006, 'log_dups_size': 0, 'ondisk_log_size': 4006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32807558, 'num_objects': 4005, 'num_object_clones': 0, 'num_object_copies': 16020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4005, 'num_write_kb': 32039, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'3983", 'reported_seq': 4058, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:16.121365+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:16.121365+0000', 'last_peered': '2024-09-15T21:05:16.121365+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:16.121365+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3983, 'log_dups_size': 0, 'ondisk_log_size': 3983, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32619142, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 15928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3982, 'num_write_kb': 31855, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4048", 'reported_seq': 4098, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:18.192746+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:18.192746+0000', 'last_peered': '2024-09-15T21:05:18.192746+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:18.192746+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4048, 'log_dups_size': 0, 'ondisk_log_size': 4048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33125644, 'num_objects': 4044, 'num_object_clones': 0, 'num_object_copies': 16176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4046, 'num_write_kb': 32364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'3988", 'reported_seq': 4066, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:16.161168+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:16.161168+0000', 'last_peered': '2024-09-15T21:05:16.161168+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:16.161168+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 3988, 'log_dups_size': 0, 'ondisk_log_size': 3988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32642316, 'num_objects': 3985, 'num_object_clones': 0, 'num_object_copies': 15940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 3986, 'num_write_kb': 31885, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:21.824 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:21.825 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4067", 'reported_seq': 4130, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:17.653601+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:17.653601+0000', 'last_peered': '2024-09-15T21:05:17.653601+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:17.653601+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4067, 'log_dups_size': 0, 'ondisk_log_size': 4067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33273100, 'num_objects': 4062, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4065, 'num_write_kb': 32508, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:21.825 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:22.576 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:22.576 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36170/260492 objects degraded (13.885%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":188,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65123,"data_bytes":533448360,"bytes_used":1719808000,"bytes_avail":384827248640,"bytes_total":386547056640,"degraded_objects":36170,"degraded_total":260492,"degraded_ratio":0.13885263271040954,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":82565,"recovering_keys_per_sec":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"write_bytes_sec":2470136,"read_op_per_sec":0,"write_op_per_sec":301},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:22.601 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:22.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:22.641 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 119 20 66327 66307 4.35257 2.26562 0.11354 0.0358823 2024-09-15T21:05:23.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:23.170 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:23.203 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-15T21:05:21.943077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":533448360,"num_objects":65123,"num_object_clones":0,"num_object_copies":260492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65150,"num_write_kb":521082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65177,"ondisk_log_size":65177,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1679500,"kb_used_data":900872,"kb_used_omap":14,"kb_used_meta":778545,"kb_avail":375807860,"statfs":{"total":386547056640,"available":384827248640,"internally_reserved":0,"allocated":922492928,"data_stored":921865128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":797230642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,6,8,10,18,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29655040,"num_objects":3620,"num_object_clones":0,"num_object_copies":14480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3620,"num_write_kb":28960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":991232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005426"},"pg_stats":[{"pgid":"1.f","version":"31'4012","reported_seq":4052,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.151204+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:18.151204+0000","last_peered":"2024-09-15T21:05:18.151204+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:18.151204+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32866304,"num_objects":4012,"num_object_clones":0,"num_object_copies":16048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4012,"num_write_kb":32096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4078","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.664510+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:17.664510+0000","last_peered":"2024-09-15T21:05:17.664510+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:17.664510+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33371404,"num_objects":4074,"num_object_clones":0,"num_object_copies":16296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4076,"num_write_kb":32604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1694}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4035","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.278675+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:21.278675+0000","last_peered":"2024-09-15T21:05:21.278675+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:21.278675+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33028742,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4089","reported_seq":4153,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.607039+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:17.607039+0000","last_peered":"2024-09-15T21:05:17.607039+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:17.607039+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4089,"log_dups_size":0,"ondisk_log_size":4089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33461516,"num_objects":4085,"num_object_clones":0,"num_object_copies":16340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":32685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4109","reported_seq":4159,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.060164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:18.060164+0000","last_peered":"2024-09-15T21:05:18.060164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:18.060164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33633548,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4038","reported_seq":4077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.077797+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:18.077797+0000","last_peered":"2024-09-15T21:05:18.077797+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:18.077797+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33079296,"num_objects":4038,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4038,"num_write_kb":32304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1722}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4109","reported_seq":4173,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.673368+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:17.673368+0000","last_peered":"2024-09-15T21:05:17.673368+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:17.673368+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4109,"log_dups_size":0,"ondisk_log_size":4109,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33608972,"num_objects":4103,"num_object_clones":0,"num_object_copies":16412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":32836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4187","reported_seq":4251,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.570472+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:17.570472+0000","last_peered":"2024-09-15T21:05:17.570472+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:17.570472+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4187,"log_dups_size":0,"ondisk_log_size":4187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34256140,"num_objects":4182,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4185,"num_write_kb":33468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1708}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4052","reported_seq":4104,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.142694+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:18.142694+0000","last_peered":"2024-09-15T21:05:18.142694+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:18.142694+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4052,"log_dups_size":0,"ondisk_log_size":4052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33122840,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4049,"num_write_kb":32369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4127","reported_seq":4203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.023093+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:21.023093+0000","last_peered":"2024-09-15T21:05:21.023093+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:21.023093+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33789196,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4019","reported_seq":5900,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.170626+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:18.170626+0000","last_peered":"2024-09-15T21:05:18.170626+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:18.170626+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32878482,"num_objects":4014,"num_object_clones":0,"num_object_copies":16056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4016,"num_write_kb":32116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":916,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4006","reported_seq":4040,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.016627+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:18.016627+0000","last_peered":"2024-09-15T21:05:18.016627+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:18.016627+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32807558,"num_objects":4005,"num_object_clones":0,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":32039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1749}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4099","reported_seq":4174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.132761+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:21.132761+0000","last_peered":"2024-09-15T21:05:21.132761+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:21.132761+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33569414,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4048","reported_seq":4098,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:18.192746+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:18.192746+0000","last_peered":"2024-09-15T21:05:18.192746+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:18.192746+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33125644,"num_objects":4044,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4046,"num_write_kb":32364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4102","reported_seq":4180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.198733+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:21.198733+0000","last_peered":"2024-09-15T21:05:21.198733+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:21.198733+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576204,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4100,"num_write_kb":32797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4067","reported_seq":4130,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:17.653601+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:17.653601+0000","last_peered":"2024-09-15T21:05:17.653601+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:17.653601+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":32508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1747}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":533448360,"num_objects":65123,"num_object_clones":0,"num_object_copies":260492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65150,"num_write_kb":521082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":7479296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":923860992,"data_stored":923860992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65177,"ondisk_log_size":65177,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479260,"kb_used_data":258764,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93892580,"statfs":{"total":96636764160,"available":96146001920,"internally_reserved":0,"allocated":264974336,"data_stored":264817386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475100,"kb_used_data":263436,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93896740,"statfs":{"total":96636764160,"available":96150261760,"internally_reserved":0,"allocated":269758464,"data_stored":269601514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249916,"kb_used_data":119724,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94121924,"statfs":{"total":96636764160,"available":96380850176,"internally_reserved":0,"allocated":122597376,"data_stored":122440426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475224,"kb_used_data":258948,"kb_used_omap":4,"kb_used_meta":216251,"kb_avail":93896616,"statfs":{"total":96636764160,"available":96150134784,"internally_reserved":0,"allocated":265162752,"data_stored":265005802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":221441742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,3,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":265256960,"data_stored":265256960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122630144,"data_stored":122630144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270053376,"data_stored":270053376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265920512,"data_stored":265920512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:23.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:05:23.639983+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0360171 2024-09-15T21:05:23.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:05:23.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 120 20 66635 66615 4.33635 2.40625 0.0380822 0.0360171 2024-09-15T21:05:24.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 121 20 66970 66950 4.32214 2.61719 0.0890282 0.0361389 2024-09-15T21:05:25.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 122 20 67310 67290 4.30849 2.65625 0.0528144 0.0362543 2024-09-15T21:05:26.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:26.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 123 20 67622 67602 4.29328 2.4375 0.0243874 0.0363786 2024-09-15T21:05:26.848 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:26.848 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:26.868 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-15T21:05:25.948688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543475368,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66374,"num_write_kb":530874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66401,"ondisk_log_size":66401,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1685128,"kb_used_data":914820,"kb_used_omap":14,"kb_used_meta":770225,"kb_avail":375802232,"statfs":{"total":386547056640,"available":384821485568,"internally_reserved":0,"allocated":936775680,"data_stored":936147880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":788710962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,16,9,6,13,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26730496,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3263,"num_write_kb":26104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009127"},"pg_stats":[{"pgid":"1.f","version":"31'4116","reported_seq":4156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.210998+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:23.210998+0000","last_peered":"2024-09-15T21:05:23.210998+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:23.210998+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4116,"log_dups_size":0,"ondisk_log_size":4116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4116,"num_write_kb":32928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4173","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.698904+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:22.698904+0000","last_peered":"2024-09-15T21:05:22.698904+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:22.698904+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34149644,"num_objects":4169,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":33364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4035","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.278675+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:21.278675+0000","last_peered":"2024-09-15T21:05:21.278675+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:21.278675+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33028742,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4192","reported_seq":4256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.631378+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:22.631378+0000","last_peered":"2024-09-15T21:05:22.631378+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:22.631378+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34305292,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4228","reported_seq":4277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.224465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:23.224465+0000","last_peered":"2024-09-15T21:05:23.224465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:23.224465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34608396,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":33805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4131","reported_seq":4170,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.169020+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:23.169020+0000","last_peered":"2024-09-15T21:05:23.169020+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:23.169020+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33841152,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":33048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4214","reported_seq":4279,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.640537+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:22.640537+0000","last_peered":"2024-09-15T21:05:22.640537+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:22.640537+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34469132,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":33676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4285","reported_seq":4349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.709086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:22.709086+0000","last_peered":"2024-09-15T21:05:22.709086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:22.709086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":34252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4150","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.203086+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:23.203086+0000","last_peered":"2024-09-15T21:05:23.203086+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:23.203086+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33925656,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4127","reported_seq":4203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.023093+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:21.023093+0000","last_peered":"2024-09-15T21:05:21.023093+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:21.023093+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33789196,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4111","reported_seq":6119,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.276350+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:23.276350+0000","last_peered":"2024-09-15T21:05:23.276350+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:23.276350+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33632146,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4106","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.239473+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:23.239473+0000","last_peered":"2024-09-15T21:05:23.239473+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:23.239473+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33626758,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":32839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4099","reported_seq":4174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.132761+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:21.132761+0000","last_peered":"2024-09-15T21:05:21.132761+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:21.132761+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33569414,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4168","reported_seq":4219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.193348+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:23.193348+0000","last_peered":"2024-09-15T21:05:23.193348+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:23.193348+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34108684,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4166,"num_write_kb":33324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1885}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4102","reported_seq":4180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.198733+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:21.198733+0000","last_peered":"2024-09-15T21:05:21.198733+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:21.198733+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576204,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4100,"num_write_kb":32797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4164","reported_seq":4227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.662645+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:22.662645+0000","last_peered":"2024-09-15T21:05:22.662645+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:22.662645+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":543475368,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66374,"num_write_kb":530874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936943616,"data_stored":936943616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66401,"ondisk_log_size":66401,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474328,"kb_used_data":266312,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93897512,"statfs":{"total":96636764160,"available":96151052288,"internally_reserved":0,"allocated":272703488,"data_stored":272546538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,12,1,0,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475100,"kb_used_data":263436,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93896740,"statfs":{"total":96636764160,"available":96150261760,"internally_reserved":0,"allocated":269758464,"data_stored":269601514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249916,"kb_used_data":119724,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94121924,"statfs":{"total":96636764160,"available":96380850176,"internally_reserved":0,"allocated":122597376,"data_stored":122440426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485784,"kb_used_data":265348,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93886056,"statfs":{"total":96636764160,"available":96139321344,"internally_reserved":0,"allocated":271716352,"data_stored":271559402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271806464,"data_stored":271806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122630144,"data_stored":122630144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270053376,"data_stored":270053376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272453632,"data_stored":272453632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:26.869 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:27.468 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:27.469 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:27.497 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-15T21:05:25.948688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543475368,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66374,"num_write_kb":530874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66401,"ondisk_log_size":66401,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1685128,"kb_used_data":914820,"kb_used_omap":14,"kb_used_meta":770225,"kb_avail":375802232,"statfs":{"total":386547056640,"available":384821485568,"internally_reserved":0,"allocated":936775680,"data_stored":936147880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":788710962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,16,9,6,13,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26730496,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3263,"num_write_kb":26104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009127"},"pg_stats":[{"pgid":"1.f","version":"31'4116","reported_seq":4156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.210998+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:23.210998+0000","last_peered":"2024-09-15T21:05:23.210998+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:23.210998+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4116,"log_dups_size":0,"ondisk_log_size":4116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4116,"num_write_kb":32928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4173","reported_seq":4235,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.698904+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:22.698904+0000","last_peered":"2024-09-15T21:05:22.698904+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:22.698904+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34149644,"num_objects":4169,"num_object_clones":0,"num_object_copies":16676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":33364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4035","reported_seq":4108,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.278675+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:21.278675+0000","last_peered":"2024-09-15T21:05:21.278675+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:21.278675+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33028742,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":32262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4192","reported_seq":4256,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.631378+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:22.631378+0000","last_peered":"2024-09-15T21:05:22.631378+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:22.631378+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34305292,"num_objects":4188,"num_object_clones":0,"num_object_copies":16752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":33509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1854}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4228","reported_seq":4277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.224465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:23.224465+0000","last_peered":"2024-09-15T21:05:23.224465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:23.224465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34608396,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":33805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4131","reported_seq":4170,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.169020+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:23.169020+0000","last_peered":"2024-09-15T21:05:23.169020+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:23.169020+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33841152,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":33048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4214","reported_seq":4279,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.640537+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:22.640537+0000","last_peered":"2024-09-15T21:05:22.640537+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:22.640537+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4214,"log_dups_size":0,"ondisk_log_size":4214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34469132,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4212,"num_write_kb":33676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1842}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4285","reported_seq":4349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.709086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:22.709086+0000","last_peered":"2024-09-15T21:05:22.709086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:22.709086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":34252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1806}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4150","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.203086+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:23.203086+0000","last_peered":"2024-09-15T21:05:23.203086+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:23.203086+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33925656,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4127","reported_seq":4203,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.023093+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:21.023093+0000","last_peered":"2024-09-15T21:05:21.023093+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:21.023093+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4127,"log_dups_size":0,"ondisk_log_size":4127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33789196,"num_objects":4125,"num_object_clones":0,"num_object_copies":16500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":32998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1790}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4111","reported_seq":6119,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.276350+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:23.276350+0000","last_peered":"2024-09-15T21:05:23.276350+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:23.276350+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33632146,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4106","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.239473+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:23.239473+0000","last_peered":"2024-09-15T21:05:23.239473+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:23.239473+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33626758,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":32839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4099","reported_seq":4174,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.132761+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:21.132761+0000","last_peered":"2024-09-15T21:05:21.132761+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:21.132761+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33569414,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":32783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1812}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4168","reported_seq":4219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.193348+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:23.193348+0000","last_peered":"2024-09-15T21:05:23.193348+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:23.193348+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34108684,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4166,"num_write_kb":33324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1885}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4102","reported_seq":4180,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:21.198733+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:21.198733+0000","last_peered":"2024-09-15T21:05:21.198733+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:21.198733+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4102,"log_dups_size":0,"ondisk_log_size":4102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33576204,"num_objects":4099,"num_object_clones":0,"num_object_copies":16396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4100,"num_write_kb":32797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4164","reported_seq":4227,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:22.662645+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:22.662645+0000","last_peered":"2024-09-15T21:05:22.662645+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:22.662645+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4162,"num_write_kb":33284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":543475368,"num_objects":66347,"num_object_clones":0,"num_object_copies":265388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":66374,"num_write_kb":530874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":936943616,"data_stored":936943616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66401,"ondisk_log_size":66401,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474328,"kb_used_data":266312,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93897512,"statfs":{"total":96636764160,"available":96151052288,"internally_reserved":0,"allocated":272703488,"data_stored":272546538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,12,1,0,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":475100,"kb_used_data":263436,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93896740,"statfs":{"total":96636764160,"available":96150261760,"internally_reserved":0,"allocated":269758464,"data_stored":269601514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051602,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249916,"kb_used_data":119724,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":94121924,"statfs":{"total":96636764160,"available":96380850176,"internally_reserved":0,"allocated":122597376,"data_stored":122440426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":133295823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,7,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485784,"kb_used_data":265348,"kb_used_omap":4,"kb_used_meta":220411,"kb_avail":93886056,"statfs":{"total":96636764160,"available":96139321344,"internally_reserved":0,"allocated":271716352,"data_stored":271559402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":225701582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271806464,"data_stored":271806464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122630144,"data_stored":122630144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":270053376,"data_stored":270053376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272453632,"data_stored":272453632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:27.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:27.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4116", 'reported_seq': 4156, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.210998+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:23.210998+0000', 'last_peered': '2024-09-15T21:05:23.210998+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:23.210998+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4116, 'log_dups_size': 0, 'ondisk_log_size': 4116, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33718272, 'num_objects': 4116, 'num_object_clones': 0, 'num_object_copies': 16464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4116, 'num_write_kb': 32928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1840}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4173", 'reported_seq': 4235, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:22.698904+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:22.698904+0000', 'last_peered': '2024-09-15T21:05:22.698904+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:22.698904+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4173, 'log_dups_size': 0, 'ondisk_log_size': 4173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34149644, 'num_objects': 4169, 'num_object_clones': 0, 'num_object_copies': 16676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4171, 'num_write_kb': 33364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1789}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4035", 'reported_seq': 4108, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:21.278675+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:21.278675+0000', 'last_peered': '2024-09-15T21:05:21.278675+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:21.278675+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4035, 'log_dups_size': 0, 'ondisk_log_size': 4035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33028742, 'num_objects': 4032, 'num_object_clones': 0, 'num_object_copies': 16128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4034, 'num_write_kb': 32262, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1774}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4192", 'reported_seq': 4256, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:22.631378+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:22.631378+0000', 'last_peered': '2024-09-15T21:05:22.631378+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:22.631378+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4192, 'log_dups_size': 0, 'ondisk_log_size': 4192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34305292, 'num_objects': 4188, 'num_object_clones': 0, 'num_object_copies': 16752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4190, 'num_write_kb': 33509, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:27.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4228", 'reported_seq': 4277, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.224465+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:23.224465+0000', 'last_peered': '2024-09-15T21:05:23.224465+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:23.224465+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4228, 'log_dups_size': 0, 'ondisk_log_size': 4228, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34608396, 'num_objects': 4225, 'num_object_clones': 0, 'num_object_copies': 16900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4226, 'num_write_kb': 33805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1914}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4131", 'reported_seq': 4170, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.169020+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:23.169020+0000', 'last_peered': '2024-09-15T21:05:23.169020+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:23.169020+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4131, 'log_dups_size': 0, 'ondisk_log_size': 4131, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33841152, 'num_objects': 4131, 'num_object_clones': 0, 'num_object_copies': 16524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4131, 'num_write_kb': 33048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1815}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4214", 'reported_seq': 4279, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:22.640537+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:22.640537+0000', 'last_peered': '2024-09-15T21:05:22.640537+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:22.640537+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4214, 'log_dups_size': 0, 'ondisk_log_size': 4214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34469132, 'num_objects': 4208, 'num_object_clones': 0, 'num_object_copies': 16832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4212, 'num_write_kb': 33676, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1842}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4285", 'reported_seq': 4349, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:22.709086+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:22.709086+0000', 'last_peered': '2024-09-15T21:05:22.709086+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:22.709086+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4285, 'log_dups_size': 0, 'ondisk_log_size': 4285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35058956, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4283, 'num_write_kb': 34252, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1806}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4150", 'reported_seq': 4205, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.203086+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:23.203086+0000', 'last_peered': '2024-09-15T21:05:23.203086+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:23.203086+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4150, 'log_dups_size': 0, 'ondisk_log_size': 4150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33925656, 'num_objects': 4142, 'num_object_clones': 0, 'num_object_copies': 16568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4147, 'num_write_kb': 33153, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1810}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:27.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4127", 'reported_seq': 4203, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:21.023093+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:21.023093+0000', 'last_peered': '2024-09-15T21:05:21.023093+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:21.023093+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4127, 'log_dups_size': 0, 'ondisk_log_size': 4127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33789196, 'num_objects': 4125, 'num_object_clones': 0, 'num_object_copies': 16500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4125, 'num_write_kb': 32998, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1790}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4111", 'reported_seq': 6119, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.276350+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:23.276350+0000', 'last_peered': '2024-09-15T21:05:23.276350+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:23.276350+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4111, 'log_dups_size': 0, 'ondisk_log_size': 4111, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33632146, 'num_objects': 4106, 'num_object_clones': 0, 'num_object_copies': 16424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1284, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4106, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4108, 'num_write_kb': 32852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 979, 'num_bytes_recovered': 7995392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4106", 'reported_seq': 4140, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.239473+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:23.239473+0000', 'last_peered': '2024-09-15T21:05:23.239473+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:23.239473+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4106, 'log_dups_size': 0, 'ondisk_log_size': 4106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33626758, 'num_objects': 4105, 'num_object_clones': 0, 'num_object_copies': 16420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4105, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4105, 'num_write_kb': 32839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4099", 'reported_seq': 4174, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:21.132761+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:21.132761+0000', 'last_peered': '2024-09-15T21:05:21.132761+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:21.132761+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4099, 'log_dups_size': 0, 'ondisk_log_size': 4099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33569414, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 16392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4098, 'num_write_kb': 32783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1812}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4168", 'reported_seq': 4219, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:23.193348+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:23.193348+0000', 'last_peered': '2024-09-15T21:05:23.193348+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:23.193348+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4168, 'log_dups_size': 0, 'ondisk_log_size': 4168, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34108684, 'num_objects': 4164, 'num_object_clones': 0, 'num_object_copies': 16656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4164, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4166, 'num_write_kb': 33324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1885}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4102", 'reported_seq': 4180, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:21.198733+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:21.198733+0000', 'last_peered': '2024-09-15T21:05:21.198733+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:21.198733+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4102, 'log_dups_size': 0, 'ondisk_log_size': 4102, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33576204, 'num_objects': 4099, 'num_object_clones': 0, 'num_object_copies': 16396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4099, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4100, 'num_write_kb': 32797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:27.501 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:27.502 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4164", 'reported_seq': 4227, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:22.662645+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:22.662645+0000', 'last_peered': '2024-09-15T21:05:22.662645+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:22.662645+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4164, 'log_dups_size': 0, 'ondisk_log_size': 4164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34067724, 'num_objects': 4159, 'num_object_clones': 0, 'num_object_copies': 16636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4162, 'num_write_kb': 33284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:27.502 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:27.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 124 20 67914 67894 4.27705 2.28125 0.0669907 0.0365152 2024-09-15T21:05:28.260 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:28.260 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36107/268824 objects degraded (13.431%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":194,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67206,"data_bytes":550512296,"bytes_used":1759625216,"bytes_avail":384787431424,"bytes_total":386547056640,"degraded_objects":36107,"degraded_total":268824,"degraded_ratio":0.1343146445257864,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":81859,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"write_bytes_sec":2811886,"read_op_per_sec":0,"write_op_per_sec":343},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:28.287 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:28.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:28.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 125 20 68216 68196 4.26171 2.35938 0.0969224 0.0366524 2024-09-15T21:05:28.898 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:28.898 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:28.921 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-15T21:05:27.949174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":550512296,"num_objects":67206,"num_object_clones":0,"num_object_copies":268824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67233,"num_write_kb":537746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67260,"ondisk_log_size":67260,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1718384,"kb_used_data":934188,"kb_used_omap":14,"kb_used_meta":784113,"kb_avail":375768976,"statfs":{"total":386547056640,"available":384787431424,"internally_reserved":0,"allocated":956608512,"data_stored":955980712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":802932274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,20,4,4,19,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33767424,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4122,"num_write_kb":32976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008814"},"pg_stats":[{"pgid":"1.f","version":"31'4116","reported_seq":4156,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.210998+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:23.210998+0000","last_peered":"2024-09-15T21:05:23.210998+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:23.210998+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4116,"log_dups_size":0,"ondisk_log_size":4116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4116,"num_write_kb":32928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1840}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4259","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.579882+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:27.579882+0000","last_peered":"2024-09-15T21:05:27.579882+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:27.579882+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854156,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1875}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4136","reported_seq":4209,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:26.331891+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:26.331891+0000","last_peered":"2024-09-15T21:05:26.331891+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:26.331891+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4136,"log_dups_size":0,"ondisk_log_size":4136,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33856134,"num_objects":4133,"num_object_clones":0,"num_object_copies":16532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4135,"num_write_kb":33070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1875}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4278","reported_seq":4342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.695447+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:27.695447+0000","last_peered":"2024-09-15T21:05:27.695447+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:27.695447+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":34197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4228","reported_seq":4277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.224465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:23.224465+0000","last_peered":"2024-09-15T21:05:23.224465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:23.224465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34608396,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":33805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1914}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4131","reported_seq":4170,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.169020+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:23.169020+0000","last_peered":"2024-09-15T21:05:23.169020+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:23.169020+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4131,"log_dups_size":0,"ondisk_log_size":4131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33841152,"num_objects":4131,"num_object_clones":0,"num_object_copies":16524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4131,"num_write_kb":33048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1815}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4326","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.752203+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:27.752203+0000","last_peered":"2024-09-15T21:05:27.752203+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:27.752203+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35386636,"num_objects":4320,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4374","reported_seq":4438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.581149+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:27.581149+0000","last_peered":"2024-09-15T21:05:27.581149+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:27.581149+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4372,"num_write_kb":34964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4150","reported_seq":4205,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.203086+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:23.203086+0000","last_peered":"2024-09-15T21:05:23.203086+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:23.203086+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33925656,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":33153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1810}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4219","reported_seq":4295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:26.082507+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:26.082507+0000","last_peered":"2024-09-15T21:05:26.082507+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:26.082507+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542860,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":33734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1882}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4111","reported_seq":6119,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.276350+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:23.276350+0000","last_peered":"2024-09-15T21:05:23.276350+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:23.276350+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33632146,"num_objects":4106,"num_object_clones":0,"num_object_copies":16424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":32852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":979,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2822}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4106","reported_seq":4140,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.239473+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:23.239473+0000","last_peered":"2024-09-15T21:05:23.239473+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:23.239473+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4106,"log_dups_size":0,"ondisk_log_size":4106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33626758,"num_objects":4105,"num_object_clones":0,"num_object_copies":16420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":32839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4184","reported_seq":4259,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:26.133802+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:26.133802+0000","last_peered":"2024-09-15T21:05:26.133802+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:26.133802+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34265734,"num_objects":4183,"num_object_clones":0,"num_object_copies":16732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4183,"num_write_kb":33463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4168","reported_seq":4219,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:23.193348+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:23.193348+0000","last_peered":"2024-09-15T21:05:23.193348+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:23.193348+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4168,"log_dups_size":0,"ondisk_log_size":4168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34108684,"num_objects":4164,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4166,"num_write_kb":33324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1885}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4181","reported_seq":4259,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:26.170342+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:26.170342+0000","last_peered":"2024-09-15T21:05:26.170342+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:26.170342+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223372,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":33429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4293","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.745882+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:27.745882+0000","last_peered":"2024-09-15T21:05:27.745882+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:27.745882+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":550512296,"num_objects":67206,"num_object_clones":0,"num_object_copies":268824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67233,"num_write_kb":537746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":7995392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":956928000,"data_stored":956928000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67260,"ondisk_log_size":67260,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":474328,"kb_used_data":266312,"kb_used_omap":4,"kb_used_meta":207995,"kb_avail":93897512,"statfs":{"total":96636764160,"available":96151052288,"internally_reserved":0,"allocated":272703488,"data_stored":272546538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":212987600},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,12,1,0,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":485780,"kb_used_data":269956,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":93886060,"statfs":{"total":96636764160,"available":96139325440,"internally_reserved":0,"allocated":276434944,"data_stored":276277994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,1,10,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":278656,"kb_used_data":126252,"kb_used_omap":4,"kb_used_meta":152379,"kb_avail":94093184,"statfs":{"total":96636764160,"available":96351420416,"internally_reserved":0,"allocated":129282048,"data_stored":129125098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":156036815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,2,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479620,"kb_used_data":271668,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93892220,"statfs":{"total":96636764160,"available":96145633280,"internally_reserved":0,"allocated":278188032,"data_stored":278031082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278339584,"data_stored":278339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129507328,"data_stored":129507328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":276627456,"data_stored":276627456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272453632,"data_stored":272453632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:29.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 126 20 68528 68508 4.24723 2.4375 0.095366 0.036772 2024-09-15T21:05:30.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 127 20 68858 68838 4.23408 2.57812 0.0571784 0.0368869 2024-09-15T21:05:31.642 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 128 20 69189 69169 4.22121 2.58594 0.0427814 0.0370005 2024-09-15T21:05:31.923 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:32.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:32.519 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:32.539 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-15T21:05:31.950674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":559810216,"num_objects":68341,"num_object_clones":0,"num_object_copies":273364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68368,"num_write_kb":546826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68395,"ondisk_log_size":68395,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1733164,"kb_used_data":953132,"kb_used_omap":14,"kb_used_meta":779953,"kb_avail":375754196,"statfs":{"total":386547056640,"available":384772296704,"internally_reserved":0,"allocated":976007168,"data_stored":975379368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":798672434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,8,9,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29573120,"num_objects":3610,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":28880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008312"},"pg_stats":[{"pgid":"1.f","version":"31'4225","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.323524+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:28.323524+0000","last_peered":"2024-09-15T21:05:28.323524+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:28.323524+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4225,"log_dups_size":0,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34611200,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4259","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.579882+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:27.579882+0000","last_peered":"2024-09-15T21:05:27.579882+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:27.579882+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854156,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1875}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4240","reported_seq":4314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.179228+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:31.179228+0000","last_peered":"2024-09-15T21:05:31.179228+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:31.179228+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34708102,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4278","reported_seq":4342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.695447+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:27.695447+0000","last_peered":"2024-09-15T21:05:27.695447+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:27.695447+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":34197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4345","reported_seq":4396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.298142+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:28.298142+0000","last_peered":"2024-09-15T21:05:28.298142+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:28.298142+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35566860,"num_objects":4342,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4236","reported_seq":4276,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.345837+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:28.345837+0000","last_peered":"2024-09-15T21:05:28.345837+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:28.345837+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34701312,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":33888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4326","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.752203+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:27.752203+0000","last_peered":"2024-09-15T21:05:27.752203+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:27.752203+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35386636,"num_objects":4320,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4374","reported_seq":4438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.581149+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:27.581149+0000","last_peered":"2024-09-15T21:05:27.581149+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:27.581149+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4372,"num_write_kb":34964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4253","reported_seq":4307,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.296646+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:28.296646+0000","last_peered":"2024-09-15T21:05:28.296646+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:28.296646+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769432,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":33977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1913}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4308","reported_seq":4384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.334338+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:31.334338+0000","last_peered":"2024-09-15T21:05:31.334338+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:31.334338+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271948,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4213","reported_seq":6355,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.310644+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:28.310644+0000","last_peered":"2024-09-15T21:05:28.310644+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:28.310644+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34467730,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4203","reported_seq":4238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.343866+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:28.343866+0000","last_peered":"2024-09-15T21:05:28.343866+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:28.343866+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34421382,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4285","reported_seq":4359,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.323718+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:31.323718+0000","last_peered":"2024-09-15T21:05:31.323718+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:31.323718+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4275","reported_seq":4326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.300684+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:28.300684+0000","last_peered":"2024-09-15T21:05:28.300684+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:28.300684+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34985228,"num_objects":4271,"num_object_clones":0,"num_object_copies":17084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":34180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4282","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.254623+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:31.254623+0000","last_peered":"2024-09-15T21:05:31.254623+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:31.254623+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050764,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4293","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.745882+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:27.745882+0000","last_peered":"2024-09-15T21:05:27.745882+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:27.745882+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":559810216,"num_objects":68341,"num_object_clones":0,"num_object_copies":273364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68368,"num_write_kb":546826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":976650240,"data_stored":976650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68395,"ondisk_log_size":68395,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484816,"kb_used_data":272640,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93887024,"statfs":{"total":96636764160,"available":96140312576,"internally_reserved":0,"allocated":279183360,"data_stored":279026410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496152,"kb_used_data":276168,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93875688,"statfs":{"total":96636764160,"available":96128704512,"internally_reserved":0,"allocated":282796032,"data_stored":282639082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272576,"kb_used_data":132656,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94099264,"statfs":{"total":96636764160,"available":96357646336,"internally_reserved":0,"allocated":135839744,"data_stored":135682794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479620,"kb_used_data":271668,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93892220,"statfs":{"total":96636764160,"available":96145633280,"internally_reserved":0,"allocated":278188032,"data_stored":278031082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278339584,"data_stored":278339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136208384,"data_stored":136208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283021312,"data_stored":283021312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279080960,"data_stored":279080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:32.540 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:32.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 129 20 69500 69480 4.20722 2.42969 0.0533918 0.0371267 2024-09-15T21:05:33.162 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:33.162 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:33.185 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2024-09-15T21:05:31.950674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":559810216,"num_objects":68341,"num_object_clones":0,"num_object_copies":273364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68368,"num_write_kb":546826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68395,"ondisk_log_size":68395,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1733164,"kb_used_data":953132,"kb_used_omap":14,"kb_used_meta":779953,"kb_avail":375754196,"statfs":{"total":386547056640,"available":384772296704,"internally_reserved":0,"allocated":976007168,"data_stored":975379368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":798672434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,8,9,10,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29573120,"num_objects":3610,"num_object_clones":0,"num_object_copies":14440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":28880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008312"},"pg_stats":[{"pgid":"1.f","version":"31'4225","reported_seq":4265,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.323524+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:28.323524+0000","last_peered":"2024-09-15T21:05:28.323524+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:28.323524+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4225,"log_dups_size":0,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34611200,"num_objects":4225,"num_object_clones":0,"num_object_copies":16900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4225,"num_write_kb":33800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4259","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.579882+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:27.579882+0000","last_peered":"2024-09-15T21:05:27.579882+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:27.579882+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34854156,"num_objects":4255,"num_object_clones":0,"num_object_copies":17020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":34052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1875}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4240","reported_seq":4314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.179228+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:31.179228+0000","last_peered":"2024-09-15T21:05:31.179228+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:31.179228+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34708102,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4278","reported_seq":4342,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.695447+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:27.695447+0000","last_peered":"2024-09-15T21:05:27.695447+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:27.695447+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4278,"log_dups_size":0,"ondisk_log_size":4278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35009804,"num_objects":4274,"num_object_clones":0,"num_object_copies":17096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":34197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1940}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4345","reported_seq":4396,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.298142+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:28.298142+0000","last_peered":"2024-09-15T21:05:28.298142+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:28.298142+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35566860,"num_objects":4342,"num_object_clones":0,"num_object_copies":17368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4236","reported_seq":4276,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.345837+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:28.345837+0000","last_peered":"2024-09-15T21:05:28.345837+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:28.345837+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34701312,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4236,"num_write_kb":33888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4326","reported_seq":4391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.752203+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:27.752203+0000","last_peered":"2024-09-15T21:05:27.752203+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:27.752203+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4326,"log_dups_size":0,"ondisk_log_size":4326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35386636,"num_objects":4320,"num_object_clones":0,"num_object_copies":17280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4374","reported_seq":4438,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.581149+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:27.581149+0000","last_peered":"2024-09-15T21:05:27.581149+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:27.581149+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4374,"log_dups_size":0,"ondisk_log_size":4374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4372,"num_write_kb":34964,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1895}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4253","reported_seq":4307,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.296646+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:28.296646+0000","last_peered":"2024-09-15T21:05:28.296646+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:28.296646+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34769432,"num_objects":4245,"num_object_clones":0,"num_object_copies":16980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":33977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1913}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4308","reported_seq":4384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.334338+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:31.334338+0000","last_peered":"2024-09-15T21:05:31.334338+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:31.334338+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271948,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4213","reported_seq":6355,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.310644+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:28.310644+0000","last_peered":"2024-09-15T21:05:28.310644+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:28.310644+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34467730,"num_objects":4208,"num_object_clones":0,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1217,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4210,"num_write_kb":33668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2991}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4203","reported_seq":4238,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.343866+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:28.343866+0000","last_peered":"2024-09-15T21:05:28.343866+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:28.343866+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4203,"log_dups_size":0,"ondisk_log_size":4203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34421382,"num_objects":4202,"num_object_clones":0,"num_object_copies":16808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4202,"num_write_kb":33615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1946}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4285","reported_seq":4359,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.323718+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:31.323718+0000","last_peered":"2024-09-15T21:05:31.323718+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:31.323718+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4275","reported_seq":4326,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:28.300684+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:28.300684+0000","last_peered":"2024-09-15T21:05:28.300684+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:28.300684+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34985228,"num_objects":4271,"num_object_clones":0,"num_object_copies":17084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":34180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1992}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4282","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.254623+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:31.254623+0000","last_peered":"2024-09-15T21:05:31.254623+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:31.254623+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050764,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4293","reported_seq":4357,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:27.745882+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:27.745882+0000","last_peered":"2024-09-15T21:05:27.745882+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:27.745882+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4293,"log_dups_size":0,"ondisk_log_size":4293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35124492,"num_objects":4288,"num_object_clones":0,"num_object_copies":17152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4291,"num_write_kb":34316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":559810216,"num_objects":68341,"num_object_clones":0,"num_object_copies":273364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":36040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68368,"num_write_kb":546826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":8544256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":976650240,"data_stored":976650240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68395,"ondisk_log_size":68395,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":484816,"kb_used_data":272640,"kb_used_omap":4,"kb_used_meta":212155,"kb_avail":93887024,"statfs":{"total":96636764160,"available":96140312576,"internally_reserved":0,"allocated":279183360,"data_stored":279026410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":217247440},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496152,"kb_used_data":276168,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93875688,"statfs":{"total":96636764160,"available":96128704512,"internally_reserved":0,"allocated":282796032,"data_stored":282639082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272576,"kb_used_data":132656,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94099264,"statfs":{"total":96636764160,"available":96357646336,"internally_reserved":0,"allocated":135839744,"data_stored":135682794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":479620,"kb_used_data":271668,"kb_used_omap":4,"kb_used_meta":207931,"kb_avail":93892220,"statfs":{"total":96636764160,"available":96145633280,"internally_reserved":0,"allocated":278188032,"data_stored":278031082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":212922062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278339584,"data_stored":278339584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136208384,"data_stored":136208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283021312,"data_stored":283021312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279080960,"data_stored":279080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4225", 'reported_seq': 4265, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.323524+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:28.323524+0000', 'last_peered': '2024-09-15T21:05:28.323524+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:28.323524+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4225, 'log_dups_size': 0, 'ondisk_log_size': 4225, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34611200, 'num_objects': 4225, 'num_object_clones': 0, 'num_object_copies': 16900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4225, 'num_write_kb': 33800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4259", 'reported_seq': 4322, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:27.579882+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:27.579882+0000', 'last_peered': '2024-09-15T21:05:27.579882+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:27.579882+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4259, 'log_dups_size': 0, 'ondisk_log_size': 4259, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34854156, 'num_objects': 4255, 'num_object_clones': 0, 'num_object_copies': 17020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4255, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4257, 'num_write_kb': 34052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 1875}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4240", 'reported_seq': 4314, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:31.179228+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:31.179228+0000', 'last_peered': '2024-09-15T21:05:31.179228+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:31.179228+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4240, 'log_dups_size': 0, 'ondisk_log_size': 4240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34708102, 'num_objects': 4237, 'num_object_clones': 0, 'num_object_copies': 16948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4239, 'num_write_kb': 33902, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1979}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4278", 'reported_seq': 4342, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:27.695447+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:27.695447+0000', 'last_peered': '2024-09-15T21:05:27.695447+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:27.695447+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4278, 'log_dups_size': 0, 'ondisk_log_size': 4278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35009804, 'num_objects': 4274, 'num_object_clones': 0, 'num_object_copies': 17096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4276, 'num_write_kb': 34197, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1940}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:33.186 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4345", 'reported_seq': 4396, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.298142+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:28.298142+0000', 'last_peered': '2024-09-15T21:05:28.298142+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:28.298142+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4345, 'log_dups_size': 0, 'ondisk_log_size': 4345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35566860, 'num_objects': 4342, 'num_object_clones': 0, 'num_object_copies': 17368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4343, 'num_write_kb': 34741, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2031}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4236", 'reported_seq': 4276, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.345837+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:28.345837+0000', 'last_peered': '2024-09-15T21:05:28.345837+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:28.345837+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4236, 'log_dups_size': 0, 'ondisk_log_size': 4236, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34701312, 'num_objects': 4236, 'num_object_clones': 0, 'num_object_copies': 16944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4236, 'num_write_kb': 33888, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1920}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4326", 'reported_seq': 4391, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:27.752203+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:27.752203+0000', 'last_peered': '2024-09-15T21:05:27.752203+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:27.752203+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4326, 'log_dups_size': 0, 'ondisk_log_size': 4326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35386636, 'num_objects': 4320, 'num_object_clones': 0, 'num_object_copies': 17280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4324, 'num_write_kb': 34572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1954}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4374", 'reported_seq': 4438, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:27.581149+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:27.581149+0000', 'last_peered': '2024-09-15T21:05:27.581149+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:27.581149+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4374, 'log_dups_size': 0, 'ondisk_log_size': 4374, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35788044, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4372, 'num_write_kb': 34964, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1895}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4253", 'reported_seq': 4307, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.296646+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:28.296646+0000', 'last_peered': '2024-09-15T21:05:28.296646+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:28.296646+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4253, 'log_dups_size': 0, 'ondisk_log_size': 4253, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34769432, 'num_objects': 4245, 'num_object_clones': 0, 'num_object_copies': 16980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4245, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4250, 'num_write_kb': 33977, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1913}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:33.187 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4308", 'reported_seq': 4384, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:31.334338+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:31.334338+0000', 'last_peered': '2024-09-15T21:05:31.334338+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:31.334338+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4308, 'log_dups_size': 0, 'ondisk_log_size': 4308, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35271948, 'num_objects': 4306, 'num_object_clones': 0, 'num_object_copies': 17224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4306, 'num_write_kb': 34446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1971}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4213", 'reported_seq': 6355, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.310644+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:28.310644+0000', 'last_peered': '2024-09-15T21:05:28.310644+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:28.310644+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4213, 'log_dups_size': 0, 'ondisk_log_size': 4213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34467730, 'num_objects': 4208, 'num_object_clones': 0, 'num_object_copies': 16832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1217, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4208, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4210, 'num_write_kb': 33668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1046, 'num_bytes_recovered': 8544256, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2991}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4203", 'reported_seq': 4238, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.343866+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:28.343866+0000', 'last_peered': '2024-09-15T21:05:28.343866+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:28.343866+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4203, 'log_dups_size': 0, 'ondisk_log_size': 4203, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34421382, 'num_objects': 4202, 'num_object_clones': 0, 'num_object_copies': 16808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4202, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4202, 'num_write_kb': 33615, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1946}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4285", 'reported_seq': 4359, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:31.323718+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:31.323718+0000', 'last_peered': '2024-09-15T21:05:31.323718+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:31.323718+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4285, 'log_dups_size': 0, 'ondisk_log_size': 4285, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35093126, 'num_objects': 4284, 'num_object_clones': 0, 'num_object_copies': 17136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4284, 'num_write_kb': 34271, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4275", 'reported_seq': 4326, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:28.300684+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:28.300684+0000', 'last_peered': '2024-09-15T21:05:28.300684+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:28.300684+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4275, 'log_dups_size': 0, 'ondisk_log_size': 4275, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34985228, 'num_objects': 4271, 'num_object_clones': 0, 'num_object_copies': 17084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4273, 'num_write_kb': 34180, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4282", 'reported_seq': 4360, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:31.254623+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:31.254623+0000', 'last_peered': '2024-09-15T21:05:31.254623+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:31.254623+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4282, 'log_dups_size': 0, 'ondisk_log_size': 4282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35050764, 'num_objects': 4279, 'num_object_clones': 0, 'num_object_copies': 17116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4280, 'num_write_kb': 34237, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:33.188 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:33.189 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4293", 'reported_seq': 4357, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:27.745882+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:27.745882+0000', 'last_peered': '2024-09-15T21:05:27.745882+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:27.745882+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4293, 'log_dups_size': 0, 'ondisk_log_size': 4293, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35124492, 'num_objects': 4288, 'num_object_clones': 0, 'num_object_copies': 17152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4291, 'num_write_kb': 34316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:33.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:33.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 130 20 69792 69772 4.1924 2.28125 0.0655686 0.0372567 2024-09-15T21:05:33.957 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:33.957 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 36040/273364 objects degraded (13.184%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":199,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68341,"data_bytes":559810216,"bytes_used":1774759936,"bytes_avail":384772296704,"bytes_total":386547056640,"degraded_objects":36040,"degraded_total":273364,"degraded_ratio":0.13183886685884022,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":88685,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":2462720,"read_op_per_sec":0,"write_op_per_sec":300},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:33.986 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:33.986 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:34.585 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:34.586 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:34.612 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-15T21:05:33.951403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":569435816,"num_objects":69516,"num_object_clones":0,"num_object_copies":278064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69543,"num_write_kb":556226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69570,"ondisk_log_size":69570,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1758336,"kb_used_data":965824,"kb_used_omap":14,"kb_used_meta":792433,"kb_avail":375729024,"statfs":{"total":386547056640,"available":384746520576,"internally_reserved":0,"allocated":989003776,"data_stored":988375976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":811451954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,7,9,13,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35987456,"num_objects":4393,"num_object_clones":0,"num_object_copies":17572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-188,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":35144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":1540096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008326"},"pg_stats":[{"pgid":"1.f","version":"31'4328","reported_seq":4369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.303401+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:33.303401+0000","last_peered":"2024-09-15T21:05:33.303401+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:33.303401+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4370","reported_seq":4431,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:32.772047+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:32.772047+0000","last_peered":"2024-09-15T21:05:32.772047+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:32.772047+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35763468,"num_objects":4366,"num_object_clones":0,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4368,"num_write_kb":34940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":1986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4240","reported_seq":4314,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.179228+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:31.179228+0000","last_peered":"2024-09-15T21:05:31.179228+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:31.179228+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34708102,"num_objects":4237,"num_object_clones":0,"num_object_copies":16948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4239,"num_write_kb":33902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1979}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4383","reported_seq":4447,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:32.722285+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:32.722285+0000","last_peered":"2024-09-15T21:05:32.722285+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:32.722285+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35869964,"num_objects":4379,"num_object_clones":0,"num_object_copies":17516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2045}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4429","reported_seq":4480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.297505+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:33.297505+0000","last_peered":"2024-09-15T21:05:33.297505+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:33.297505+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36254988,"num_objects":4426,"num_object_clones":0,"num_object_copies":17704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":35413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4343","reported_seq":4381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.434116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:33.434116+0000","last_peered":"2024-09-15T21:05:33.434116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:33.434116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35577856,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4424","reported_seq":4489,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:32.747844+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:32.747844+0000","last_peered":"2024-09-15T21:05:32.747844+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:32.747844+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4424,"log_dups_size":0,"ondisk_log_size":4424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36189452,"num_objects":4418,"num_object_clones":0,"num_object_copies":17672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":35356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2052}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4482","reported_seq":4546,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:32.622049+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:32.622049+0000","last_peered":"2024-09-15T21:05:32.622049+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:32.622049+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36672780,"num_objects":4477,"num_object_clones":0,"num_object_copies":17908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":35828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2003}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4342","reported_seq":4397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.329840+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:33.329840+0000","last_peered":"2024-09-15T21:05:33.329840+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:33.329840+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35498520,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":34689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4308","reported_seq":4384,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.334338+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:31.334338+0000","last_peered":"2024-09-15T21:05:31.334338+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:31.334338+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4308,"log_dups_size":0,"ondisk_log_size":4308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35271948,"num_objects":4306,"num_object_clones":0,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1971}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4309","reported_seq":6567,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.394232+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:33.394232+0000","last_peered":"2024-09-15T21:05:33.394232+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:33.394232+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35254162,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4288","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.423544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:33.423544+0000","last_peered":"2024-09-15T21:05:33.423544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:33.423544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35117702,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2031}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4285","reported_seq":4359,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.323718+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:31.323718+0000","last_peered":"2024-09-15T21:05:31.323718+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:31.323718+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4284,"num_write_kb":34271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4384","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.408777+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:33.408777+0000","last_peered":"2024-09-15T21:05:33.408777+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:33.408777+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4382,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2101}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4282","reported_seq":4360,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:31.254623+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:31.254623+0000","last_peered":"2024-09-15T21:05:31.254623+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:31.254623+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35050764,"num_objects":4279,"num_object_clones":0,"num_object_copies":17116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":34237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4373","reported_seq":4437,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:32.545362+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:32.545362+0000","last_peered":"2024-09-15T21:05:32.545362+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:32.545362+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35779852,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4371,"num_write_kb":34956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2053}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":569435816,"num_objects":69516,"num_object_clones":0,"num_object_copies":278064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69543,"num_write_kb":556226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":989667328,"data_stored":989667328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69570,"ondisk_log_size":69570,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499552,"kb_used_data":279056,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93872288,"statfs":{"total":96636764160,"available":96125222912,"internally_reserved":0,"allocated":285753344,"data_stored":285596394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":496152,"kb_used_data":276168,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93875688,"statfs":{"total":96636764160,"available":96128704512,"internally_reserved":0,"allocated":282796032,"data_stored":282639082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":225245637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051604,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":272576,"kb_used_data":132656,"kb_used_omap":4,"kb_used_meta":139899,"kb_avail":94099264,"statfs":{"total":96636764160,"available":96357646336,"internally_reserved":0,"allocated":135839744,"data_stored":135682794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":143257295},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490056,"kb_used_data":277944,"kb_used_omap":4,"kb_used_meta":212091,"kb_avail":93881784,"statfs":{"total":96636764160,"available":96134946816,"internally_reserved":0,"allocated":284614656,"data_stored":284457706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":217181902},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":284823552,"data_stored":284823552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136208384,"data_stored":136208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283021312,"data_stored":283021312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285614080,"data_stored":285614080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:34.645 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 131 20 70068 70048 4.17686 2.15625 0.116398 0.037392 2024-09-15T21:05:35.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 132 20 70398 70378 4.16475 2.57812 0.0796089 0.0375042 2024-09-15T21:05:36.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 133 20 70722 70702 4.15246 2.53125 0.0591081 0.0376143 2024-09-15T21:05:37.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:37.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 134 20 71043 71023 4.14017 2.50781 0.0606389 0.0377248 2024-09-15T21:05:38.208 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:38.208 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:38.220 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-15T21:05:37.952625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576743080,"num_objects":70408,"num_object_clones":0,"num_object_copies":281632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":563362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70462,"ondisk_log_size":70462,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1808784,"kb_used_data":984336,"kb_used_omap":14,"kb_used_meta":824369,"kb_avail":375678576,"statfs":{"total":386547056640,"available":384694861824,"internally_reserved":0,"allocated":1007960064,"data_stored":1007332264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":844154418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,9,7,13,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33267712,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":32488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003937"},"pg_stats":[{"pgid":"1.f","version":"31'4328","reported_seq":4369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.303401+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:33.303401+0000","last_peered":"2024-09-15T21:05:33.303401+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:33.303401+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4469","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.757504+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:37.757504+0000","last_peered":"2024-09-15T21:05:37.757504+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:37.757504+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4325","reported_seq":4399,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:35.992752+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:35.992752+0000","last_peered":"2024-09-15T21:05:35.992752+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:35.992752+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35404422,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2064}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4467","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.712981+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:37.712981+0000","last_peered":"2024-09-15T21:05:37.712981+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:37.712981+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558092,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4429","reported_seq":4480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.297505+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:33.297505+0000","last_peered":"2024-09-15T21:05:33.297505+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:33.297505+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36254988,"num_objects":4426,"num_object_clones":0,"num_object_copies":17704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":35413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4343","reported_seq":4381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.434116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:33.434116+0000","last_peered":"2024-09-15T21:05:33.434116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:33.434116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35577856,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4530","reported_seq":4595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.753025+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:37.753025+0000","last_peered":"2024-09-15T21:05:37.753025+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:37.753025+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":36204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4579","reported_seq":4643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.653856+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:37.653856+0000","last_peered":"2024-09-15T21:05:37.653856+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:37.653856+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4342","reported_seq":4397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.329840+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:33.329840+0000","last_peered":"2024-09-15T21:05:33.329840+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:33.329840+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35498520,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":34689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4414","reported_seq":4490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.334951+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:36.334951+0000","last_peered":"2024-09-15T21:05:36.334951+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:36.334951+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4309","reported_seq":6567,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.394232+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:33.394232+0000","last_peered":"2024-09-15T21:05:33.394232+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:33.394232+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35254162,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4288","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.423544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:33.423544+0000","last_peered":"2024-09-15T21:05:33.423544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:33.423544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35117702,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2031}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4392","reported_seq":4467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.258701+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:36.258701+0000","last_peered":"2024-09-15T21:05:36.258701+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:36.258701+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35969670,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4384","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.408777+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:33.408777+0000","last_peered":"2024-09-15T21:05:33.408777+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:33.408777+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4382,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2101}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4401","reported_seq":4478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.348583+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:36.348583+0000","last_peered":"2024-09-15T21:05:36.348583+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:36.348583+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.750269+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:37.750269+0000","last_peered":"2024-09-15T21:05:37.750269+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:37.750269+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508940,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":576743080,"num_objects":70408,"num_object_clones":0,"num_object_copies":281632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":563362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008824320,"data_stored":1008824320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70462,"ondisk_log_size":70462,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499552,"kb_used_data":279056,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93872288,"statfs":{"total":96636764160,"available":96125222912,"internally_reserved":0,"allocated":285753344,"data_stored":285596394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489688,"kb_used_data":282184,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93882152,"statfs":{"total":96636764160,"available":96135323648,"internally_reserved":0,"allocated":288956416,"data_stored":288799466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301088,"kb_used_data":138960,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94070752,"statfs":{"total":96636764160,"available":96328450048,"internally_reserved":0,"allocated":142295040,"data_stored":142138090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518456,"kb_used_data":284136,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93853384,"statfs":{"total":96636764160,"available":96105865216,"internally_reserved":0,"allocated":290955264,"data_stored":290798314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":291155968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142733312,"data_stored":142733312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289320960,"data_stored":289320960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285614080,"data_stored":285614080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:38.222 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:38.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 135 20 71305 71285 4.12467 2.04688 0.0813912 0.03786 2024-09-15T21:05:38.858 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:38.858 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:38.886 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-15T21:05:37.952625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576743080,"num_objects":70408,"num_object_clones":0,"num_object_copies":281632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":563362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70462,"ondisk_log_size":70462,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1808784,"kb_used_data":984336,"kb_used_omap":14,"kb_used_meta":824369,"kb_avail":375678576,"statfs":{"total":386547056640,"available":384694861824,"internally_reserved":0,"allocated":1007960064,"data_stored":1007332264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":844154418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,9,7,13,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33267712,"num_objects":4061,"num_object_clones":0,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4061,"num_write_kb":32488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003937"},"pg_stats":[{"pgid":"1.f","version":"31'4328","reported_seq":4369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.303401+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:33.303401+0000","last_peered":"2024-09-15T21:05:33.303401+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:33.303401+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4328,"log_dups_size":0,"ondisk_log_size":4328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":4328,"num_object_clones":0,"num_object_copies":17312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4328,"num_write_kb":34624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4469","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.757504+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:37.757504+0000","last_peered":"2024-09-15T21:05:37.757504+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:37.757504+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4325","reported_seq":4399,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:35.992752+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:35.992752+0000","last_peered":"2024-09-15T21:05:35.992752+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:35.992752+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35404422,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2064}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4467","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.712981+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:37.712981+0000","last_peered":"2024-09-15T21:05:37.712981+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:37.712981+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558092,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4429","reported_seq":4480,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.297505+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:33.297505+0000","last_peered":"2024-09-15T21:05:33.297505+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:33.297505+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36254988,"num_objects":4426,"num_object_clones":0,"num_object_copies":17704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4427,"num_write_kb":35413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4343","reported_seq":4381,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.434116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:33.434116+0000","last_peered":"2024-09-15T21:05:33.434116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:33.434116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35577856,"num_objects":4343,"num_object_clones":0,"num_object_copies":17372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4343,"num_write_kb":34744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2027}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4530","reported_seq":4595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.753025+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:37.753025+0000","last_peered":"2024-09-15T21:05:37.753025+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:37.753025+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":36204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4579","reported_seq":4643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.653856+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:37.653856+0000","last_peered":"2024-09-15T21:05:37.653856+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:37.653856+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4342","reported_seq":4397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.329840+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:33.329840+0000","last_peered":"2024-09-15T21:05:33.329840+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:33.329840+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4342,"log_dups_size":0,"ondisk_log_size":4342,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35498520,"num_objects":4334,"num_object_clones":0,"num_object_copies":17336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4339,"num_write_kb":34689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2002}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4414","reported_seq":4490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.334951+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:36.334951+0000","last_peered":"2024-09-15T21:05:36.334951+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:36.334951+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4309","reported_seq":6567,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.394232+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:33.394232+0000","last_peered":"2024-09-15T21:05:33.394232+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:33.394232+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4309,"log_dups_size":0,"ondisk_log_size":4309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35254162,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":34436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1104,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3145}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4288","reported_seq":4322,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.423544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:33.423544+0000","last_peered":"2024-09-15T21:05:33.423544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:33.423544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35117702,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4287,"num_write_kb":34295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2031}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4392","reported_seq":4467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.258701+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:36.258701+0000","last_peered":"2024-09-15T21:05:36.258701+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:36.258701+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35969670,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4384","reported_seq":4436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:33.408777+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:33.408777+0000","last_peered":"2024-09-15T21:05:33.408777+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:33.408777+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35878156,"num_objects":4380,"num_object_clones":0,"num_object_copies":17520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4382,"num_write_kb":35052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2101}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4401","reported_seq":4478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.348583+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:36.348583+0000","last_peered":"2024-09-15T21:05:36.348583+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:36.348583+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.750269+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:37.750269+0000","last_peered":"2024-09-15T21:05:37.750269+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:37.750269+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508940,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":576743080,"num_objects":70408,"num_object_clones":0,"num_object_copies":281632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70435,"num_write_kb":563362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":9019392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1008824320,"data_stored":1008824320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70462,"ondisk_log_size":70462,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":499552,"kb_used_data":279056,"kb_used_omap":4,"kb_used_meta":220475,"kb_avail":93872288,"statfs":{"total":96636764160,"available":96125222912,"internally_reserved":0,"allocated":285753344,"data_stored":285596394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":225767120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489688,"kb_used_data":282184,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93882152,"statfs":{"total":96636764160,"available":96135323648,"internally_reserved":0,"allocated":288956416,"data_stored":288799466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301088,"kb_used_data":138960,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94070752,"statfs":{"total":96636764160,"available":96328450048,"internally_reserved":0,"allocated":142295040,"data_stored":142138090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518456,"kb_used_data":284136,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93853384,"statfs":{"total":96636764160,"available":96105865216,"internally_reserved":0,"allocated":290955264,"data_stored":290798314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":291155968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142733312,"data_stored":142733312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289320960,"data_stored":289320960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285614080,"data_stored":285614080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4328", 'reported_seq': 4369, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.303401+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:33.303401+0000', 'last_peered': '2024-09-15T21:05:33.303401+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:33.303401+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4328, 'log_dups_size': 0, 'ondisk_log_size': 4328, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35454976, 'num_objects': 4328, 'num_object_clones': 0, 'num_object_copies': 17312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4328, 'num_write_kb': 34624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2052}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4469", 'reported_seq': 4531, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:37.757504+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:37.757504+0000', 'last_peered': '2024-09-15T21:05:37.757504+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:37.757504+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4469, 'log_dups_size': 0, 'ondisk_log_size': 4469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36574476, 'num_objects': 4465, 'num_object_clones': 0, 'num_object_copies': 17860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4467, 'num_write_kb': 35732, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:38.887 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4325", 'reported_seq': 4399, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:35.992752+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:35.992752+0000', 'last_peered': '2024-09-15T21:05:35.992752+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:35.992752+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4325, 'log_dups_size': 0, 'ondisk_log_size': 4325, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35404422, 'num_objects': 4322, 'num_object_clones': 0, 'num_object_copies': 17288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4324, 'num_write_kb': 34582, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2064}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4467", 'reported_seq': 4531, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:37.712981+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:37.712981+0000', 'last_peered': '2024-09-15T21:05:37.712981+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:37.712981+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4467, 'log_dups_size': 0, 'ondisk_log_size': 4467, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36558092, 'num_objects': 4463, 'num_object_clones': 0, 'num_object_copies': 17852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4463, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4465, 'num_write_kb': 35709, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2129}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4429", 'reported_seq': 4480, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.297505+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:33.297505+0000', 'last_peered': '2024-09-15T21:05:33.297505+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:33.297505+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4429, 'log_dups_size': 0, 'ondisk_log_size': 4429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36254988, 'num_objects': 4426, 'num_object_clones': 0, 'num_object_copies': 17704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4427, 'num_write_kb': 35413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2115}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4343", 'reported_seq': 4381, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.434116+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:33.434116+0000', 'last_peered': '2024-09-15T21:05:33.434116+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:33.434116+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4343, 'log_dups_size': 0, 'ondisk_log_size': 4343, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35577856, 'num_objects': 4343, 'num_object_clones': 0, 'num_object_copies': 17372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4343, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4343, 'num_write_kb': 34744, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2027}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4530", 'reported_seq': 4595, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:37.753025+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:37.753025+0000', 'last_peered': '2024-09-15T21:05:37.753025+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:37.753025+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4530, 'log_dups_size': 0, 'ondisk_log_size': 4530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37057804, 'num_objects': 4524, 'num_object_clones': 0, 'num_object_copies': 18096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4528, 'num_write_kb': 36204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2158}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:38.888 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4579", 'reported_seq': 4643, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:37.653856+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:37.653856+0000', 'last_peered': '2024-09-15T21:05:37.653856+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:37.653856+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4579, 'log_dups_size': 0, 'ondisk_log_size': 4579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37467404, 'num_objects': 4574, 'num_object_clones': 0, 'num_object_copies': 18296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4577, 'num_write_kb': 36604, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2100}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4342", 'reported_seq': 4397, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.329840+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:33.329840+0000', 'last_peered': '2024-09-15T21:05:33.329840+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:33.329840+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4342, 'log_dups_size': 0, 'ondisk_log_size': 4342, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35498520, 'num_objects': 4334, 'num_object_clones': 0, 'num_object_copies': 17336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4339, 'num_write_kb': 34689, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2002}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4414", 'reported_seq': 4490, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:36.334951+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:36.334951+0000', 'last_peered': '2024-09-15T21:05:36.334951+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:36.334951+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4414, 'log_dups_size': 0, 'ondisk_log_size': 4414, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36140300, 'num_objects': 4412, 'num_object_clones': 0, 'num_object_copies': 17648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4412, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4412, 'num_write_kb': 35294, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2077}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:38.889 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4309", 'reported_seq': 6567, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.394232+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:33.394232+0000', 'last_peered': '2024-09-15T21:05:33.394232+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:33.394232+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4309, 'log_dups_size': 0, 'ondisk_log_size': 4309, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35254162, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1159, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4306, 'num_write_kb': 34436, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1104, 'num_bytes_recovered': 9019392, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3145}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4288", 'reported_seq': 4322, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.423544+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:33.423544+0000', 'last_peered': '2024-09-15T21:05:33.423544+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:33.423544+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4288, 'log_dups_size': 0, 'ondisk_log_size': 4288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35117702, 'num_objects': 4287, 'num_object_clones': 0, 'num_object_copies': 17148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4287, 'num_write_kb': 34295, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2031}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4392", 'reported_seq': 4467, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:36.258701+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:36.258701+0000', 'last_peered': '2024-09-15T21:05:36.258701+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:36.258701+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4392, 'log_dups_size': 0, 'ondisk_log_size': 4392, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35969670, 'num_objects': 4391, 'num_object_clones': 0, 'num_object_copies': 17564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4391, 'num_write_kb': 35127, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2105}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4384", 'reported_seq': 4436, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:33.408777+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:33.408777+0000', 'last_peered': '2024-09-15T21:05:33.408777+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:33.408777+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4384, 'log_dups_size': 0, 'ondisk_log_size': 4384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35878156, 'num_objects': 4380, 'num_object_clones': 0, 'num_object_copies': 17520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4382, 'num_write_kb': 35052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2101}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4401", 'reported_seq': 4478, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:36.348583+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:36.348583+0000', 'last_peered': '2024-09-15T21:05:36.348583+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:36.348583+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4401, 'log_dups_size': 0, 'ondisk_log_size': 4401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36025612, 'num_objects': 4398, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4399, 'num_write_kb': 35189, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2117}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:38.890 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4462", 'reported_seq': 4526, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:37.750269+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:37.750269+0000', 'last_peered': '2024-09-15T21:05:37.750269+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:37.750269+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4462, 'log_dups_size': 0, 'ondisk_log_size': 4462, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36508940, 'num_objects': 4457, 'num_object_clones': 0, 'num_object_copies': 17828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4460, 'num_write_kb': 35668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:38.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:39.631 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:39.631 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35982/281632 objects degraded (12.776%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":205,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70408,"data_bytes":576743080,"bytes_used":1852194816,"bytes_avail":384694861824,"bytes_total":386547056640,"degraded_objects":35982,"degraded_total":281632,"degraded_ratio":0.12776247017384387,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":85305,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"write_bytes_sec":2771400,"read_op_per_sec":0,"write_op_per_sec":338},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:39.646 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 136 20 71588 71568 4.1106 2.21094 0.0648318 0.0379956 2024-09-15T21:05:39.671 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:39.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:40.287 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:40.288 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:40.321 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-15T21:05:39.953551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":582182568,"num_objects":71072,"num_object_clones":0,"num_object_copies":284288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71099,"num_write_kb":568674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71126,"ondisk_log_size":71126,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1837104,"kb_used_data":990448,"kb_used_omap":14,"kb_used_meta":846577,"kb_avail":375650256,"statfs":{"total":386547056640,"available":384665862144,"internally_reserved":0,"allocated":1014218752,"data_stored":1013590952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":866895410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,11,8,9,9,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31670272,"num_objects":3866,"num_object_clones":0,"num_object_copies":15464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3866,"num_write_kb":30928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":1515520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004377"},"pg_stats":[{"pgid":"1.f","version":"31'4428","reported_seq":4469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.297012+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:38.297012+0000","last_peered":"2024-09-15T21:05:38.297012+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:38.297012+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36274176,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4428,"num_write_kb":35424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4469","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.757504+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:37.757504+0000","last_peered":"2024-09-15T21:05:37.757504+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:37.757504+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4325","reported_seq":4399,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:35.992752+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:35.992752+0000","last_peered":"2024-09-15T21:05:35.992752+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:35.992752+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35404422,"num_objects":4322,"num_object_clones":0,"num_object_copies":17288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4324,"num_write_kb":34582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2064}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4467","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.712981+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:37.712981+0000","last_peered":"2024-09-15T21:05:37.712981+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:37.712981+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558092,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4528","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.492170+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:38.492170+0000","last_peered":"2024-09-15T21:05:38.492170+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:38.492170+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37065996,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4433","reported_seq":4472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.405788+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:38.405788+0000","last_peered":"2024-09-15T21:05:38.405788+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:38.405788+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4433,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4530","reported_seq":4595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.753025+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:37.753025+0000","last_peered":"2024-09-15T21:05:37.753025+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:37.753025+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":36204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4579","reported_seq":4643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.653856+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:37.653856+0000","last_peered":"2024-09-15T21:05:37.653856+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:37.653856+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4435","reported_seq":4487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.506956+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:38.506956+0000","last_peered":"2024-09-15T21:05:38.506956+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:38.506956+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36260376,"num_objects":4427,"num_object_clones":0,"num_object_copies":17708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4414","reported_seq":4490,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.334951+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:36.334951+0000","last_peered":"2024-09-15T21:05:36.334951+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:36.334951+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36140300,"num_objects":4412,"num_object_clones":0,"num_object_copies":17648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4412,"num_write_kb":35294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2077}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4404","reported_seq":6782,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.464154+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:38.464154+0000","last_peered":"2024-09-15T21:05:38.464154+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:38.464154+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36032402,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4382","reported_seq":4417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.430677+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:38.430677+0000","last_peered":"2024-09-15T21:05:38.430677+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:38.430677+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35887750,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4392","reported_seq":4467,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.258701+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:36.258701+0000","last_peered":"2024-09-15T21:05:36.258701+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:36.258701+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35969670,"num_objects":4391,"num_object_clones":0,"num_object_copies":17564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4391,"num_write_kb":35127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2105}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4477","reported_seq":4529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.452850+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:38.452850+0000","last_peered":"2024-09-15T21:05:38.452850+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:38.452850+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36640012,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4475,"num_write_kb":35796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4401","reported_seq":4478,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:36.348583+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:36.348583+0000","last_peered":"2024-09-15T21:05:36.348583+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:36.348583+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4401,"log_dups_size":0,"ondisk_log_size":4401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4399,"num_write_kb":35189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2117}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.750269+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:37.750269+0000","last_peered":"2024-09-15T21:05:37.750269+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:37.750269+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508940,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":582182568,"num_objects":71072,"num_object_clones":0,"num_object_copies":284288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71099,"num_write_kb":568674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1015156736,"data_stored":1015156736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71126,"ondisk_log_size":71126,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527872,"kb_used_data":285168,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93843968,"statfs":{"total":96636764160,"available":96096223232,"internally_reserved":0,"allocated":292012032,"data_stored":291855082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":248508112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":489688,"kb_used_data":282184,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93882152,"statfs":{"total":96636764160,"available":96135323648,"internally_reserved":0,"allocated":288956416,"data_stored":288799466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,0,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301088,"kb_used_data":138960,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94070752,"statfs":{"total":96636764160,"available":96328450048,"internally_reserved":0,"allocated":142295040,"data_stored":142138090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,5,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518456,"kb_used_data":284136,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93853384,"statfs":{"total":96636764160,"available":96105865216,"internally_reserved":0,"allocated":290955264,"data_stored":290798314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":291155968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142733312,"data_stored":142733312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":289320960,"data_stored":289320960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291946496,"data_stored":291946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:40.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 137 20 71891 71871 4.09787 2.36719 0.0965581 0.0381196 2024-09-15T21:05:41.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 138 20 72208 72188 4.08612 2.47656 0.0493819 0.0382249 2024-09-15T21:05:42.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 139 20 72523 72503 4.07443 2.46094 0.0440645 0.0383382 2024-09-15T21:05:43.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:43.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:05:43.645298+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.038451 2024-09-15T21:05:43.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:05:43.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 140 20 72816 72796 4.06167 2.28906 0.0813662 0.038451 2024-09-15T21:05:43.953 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:43.953 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:43.986 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-15T21:05:41.954424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585148072,"num_objects":71434,"num_object_clones":0,"num_object_copies":285736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71461,"num_write_kb":571570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71488,"ondisk_log_size":71488,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1871364,"kb_used_data":1002500,"kb_used_omap":14,"kb_used_meta":868785,"kb_avail":375615996,"statfs":{"total":386547056640,"available":384630779904,"internally_reserved":0,"allocated":1026560000,"data_stored":1025932200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":889636402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,5,8,5,6,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28573696,"num_objects":3488,"num_object_clones":0,"num_object_copies":13952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3488,"num_write_kb":27904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":966656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004699"},"pg_stats":[{"pgid":"1.f","version":"31'4428","reported_seq":4469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.297012+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:38.297012+0000","last_peered":"2024-09-15T21:05:38.297012+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:38.297012+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36274176,"num_objects":4428,"num_object_clones":0,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4428,"num_write_kb":35424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2152}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4469","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.757504+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:37.757504+0000","last_peered":"2024-09-15T21:05:37.757504+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:37.757504+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4467,"num_write_kb":35732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4408","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.181574+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:41.181574+0000","last_peered":"2024-09-15T21:05:41.181574+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:41.181574+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084358,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2147}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4467","reported_seq":4531,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.712981+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:37.712981+0000","last_peered":"2024-09-15T21:05:37.712981+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:37.712981+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36558092,"num_objects":4463,"num_object_clones":0,"num_object_copies":17852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4465,"num_write_kb":35709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2129}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4528","reported_seq":4579,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.492170+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:38.492170+0000","last_peered":"2024-09-15T21:05:38.492170+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:38.492170+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37065996,"num_objects":4525,"num_object_clones":0,"num_object_copies":18100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":36205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2214}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4433","reported_seq":4472,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.405788+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:38.405788+0000","last_peered":"2024-09-15T21:05:38.405788+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:38.405788+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4433,"num_write_kb":35464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4530","reported_seq":4595,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.753025+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:37.753025+0000","last_peered":"2024-09-15T21:05:37.753025+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:37.753025+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4530,"log_dups_size":0,"ondisk_log_size":4530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37057804,"num_objects":4524,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":36204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2158}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4579","reported_seq":4643,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.653856+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:37.653856+0000","last_peered":"2024-09-15T21:05:37.653856+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:37.653856+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":36604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2100}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4435","reported_seq":4487,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.506956+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:38.506956+0000","last_peered":"2024-09-15T21:05:38.506956+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:38.506956+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36260376,"num_objects":4427,"num_object_clones":0,"num_object_copies":17708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":35433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2095}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4493","reported_seq":4569,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.320574+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:41.320574+0000","last_peered":"2024-09-15T21:05:41.320574+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:41.320574+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36787468,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4404","reported_seq":6782,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.464154+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:38.464154+0000","last_peered":"2024-09-15T21:05:38.464154+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:38.464154+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36032402,"num_objects":4399,"num_object_clones":0,"num_object_copies":17596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1099,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4401,"num_write_kb":35196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1164,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4382","reported_seq":4417,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.430677+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:38.430677+0000","last_peered":"2024-09-15T21:05:38.430677+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:38.430677+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35887750,"num_objects":4381,"num_object_clones":0,"num_object_copies":17524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":35047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2125}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4493","reported_seq":4568,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.262454+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:41.262454+0000","last_peered":"2024-09-15T21:05:41.262454+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:41.262454+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36797062,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4477","reported_seq":4529,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:38.452850+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:38.452850+0000","last_peered":"2024-09-15T21:05:38.452850+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:38.452850+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36640012,"num_objects":4473,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4475,"num_write_kb":35796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4500","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.342917+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:41.342917+0000","last_peered":"2024-09-15T21:05:41.342917+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:41.342917+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836620,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4498,"num_write_kb":35981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4462","reported_seq":4526,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:37.750269+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:37.750269+0000","last_peered":"2024-09-15T21:05:37.750269+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:37.750269+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36508940,"num_objects":4457,"num_object_clones":0,"num_object_copies":17828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":35668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":585148072,"num_objects":71434,"num_object_clones":0,"num_object_copies":285736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71461,"num_write_kb":571570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1188,"num_bytes_recovered":9510912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1027657728,"data_stored":1027657728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71488,"ondisk_log_size":71488,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":527872,"kb_used_data":285168,"kb_used_omap":4,"kb_used_meta":242683,"kb_avail":93843968,"statfs":{"total":96636764160,"available":96096223232,"internally_reserved":0,"allocated":292012032,"data_stored":291855082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":248508112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517800,"kb_used_data":288088,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93854040,"statfs":{"total":96636764160,"available":96106536960,"internally_reserved":0,"allocated":295002112,"data_stored":294845162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307236,"kb_used_data":145108,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94064604,"statfs":{"total":96636764160,"available":96322154496,"internally_reserved":0,"allocated":148590592,"data_stored":148433642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":518456,"kb_used_data":284136,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93853384,"statfs":{"total":96636764160,"available":96105865216,"internally_reserved":0,"allocated":290955264,"data_stored":290798314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":239922894},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291155968,"data_stored":291155968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149118976,"data_stored":149118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295436288,"data_stored":295436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":291946496,"data_stored":291946496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:43.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:44.601 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:44.602 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:44.634 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-15T21:05:43.955303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594396840,"num_objects":72563,"num_object_clones":0,"num_object_copies":290252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72590,"num_write_kb":580602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72617,"ondisk_log_size":72617,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1875040,"kb_used_data":1014496,"kb_used_omap":14,"kb_used_meta":860465,"kb_avail":375612320,"statfs":{"total":386547056640,"available":384627015680,"internally_reserved":0,"allocated":1038843904,"data_stored":1038216104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":881116722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,16,3,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34586624,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":33776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"1.f","version":"31'4522","reported_seq":4562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.518536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:43.518536+0000","last_peered":"2024-09-15T21:05:43.518536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:43.518536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4559","reported_seq":4622,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.637387+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:42.637387+0000","last_peered":"2024-09-15T21:05:42.637387+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:42.637387+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37311756,"num_objects":4555,"num_object_clones":0,"num_object_copies":18220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4557,"num_write_kb":36452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4408","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.181574+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:41.181574+0000","last_peered":"2024-09-15T21:05:41.181574+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:41.181574+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084358,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2147}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4543","reported_seq":4607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.753500+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:42.753500+0000","last_peered":"2024-09-15T21:05:42.753500+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:42.753500+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37180684,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4541,"num_write_kb":36317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4637","reported_seq":4688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.490597+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:43.490597+0000","last_peered":"2024-09-15T21:05:43.490597+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:43.490597+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958924,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":37077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4527","reported_seq":4566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.375449+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:43.375449+0000","last_peered":"2024-09-15T21:05:43.375449+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:43.375449+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37085184,"num_objects":4527,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.751204+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:42.751204+0000","last_peered":"2024-09-15T21:05:42.751204+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:42.751204+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2258}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4658","reported_seq":4722,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.795858+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:42.795858+0000","last_peered":"2024-09-15T21:05:42.795858+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:42.795858+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114572,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4541","reported_seq":4596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.370014+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:43.370014+0000","last_peered":"2024-09-15T21:05:43.370014+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:43.370014+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37128728,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4493","reported_seq":4569,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.320574+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:41.320574+0000","last_peered":"2024-09-15T21:05:41.320574+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:41.320574+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36787468,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4514","reported_seq":7009,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.537050+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:43.537050+0000","last_peered":"2024-09-15T21:05:43.537050+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:43.537050+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36933522,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4469","reported_seq":4502,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.445216+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:43.445216+0000","last_peered":"2024-09-15T21:05:43.445216+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:43.445216+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36600454,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":35743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2212}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4493","reported_seq":4568,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.262454+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:41.262454+0000","last_peered":"2024-09-15T21:05:41.262454+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:41.262454+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36797062,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4561","reported_seq":4613,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.350351+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:43.350351+0000","last_peered":"2024-09-15T21:05:43.350351+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:43.350351+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328140,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4500","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.342917+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:41.342917+0000","last_peered":"2024-09-15T21:05:41.342917+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:41.342917+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836620,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4498,"num_write_kb":35981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4562","reported_seq":4626,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.789646+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:42.789646+0000","last_peered":"2024-09-15T21:05:42.789646+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:42.789646+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328140,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4560,"num_write_kb":36468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594396840,"num_objects":72563,"num_object_clones":0,"num_object_copies":290252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72590,"num_write_kb":580602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1039986688,"data_stored":1039986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72617,"ondisk_log_size":72617,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521468,"kb_used_data":291244,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93850372,"statfs":{"total":96636764160,"available":96102780928,"internally_reserved":0,"allocated":298233856,"data_stored":298076906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,1,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517800,"kb_used_data":288088,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93854040,"statfs":{"total":96636764160,"available":96106536960,"internally_reserved":0,"allocated":295002112,"data_stored":294845162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307236,"kb_used_data":145108,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94064604,"statfs":{"total":96636764160,"available":96322154496,"internally_reserved":0,"allocated":148590592,"data_stored":148433642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528536,"kb_used_data":290056,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93843304,"statfs":{"total":96636764160,"available":96095543296,"internally_reserved":0,"allocated":297017344,"data_stored":296860394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297287680,"data_stored":297287680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149118976,"data_stored":149118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295436288,"data_stored":295436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298143744,"data_stored":298143744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4522", 'reported_seq': 4562, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.518536+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:43.518536+0000', 'last_peered': '2024-09-15T21:05:43.518536+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:43.518536+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4522, 'log_dups_size': 0, 'ondisk_log_size': 4522, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37044224, 'num_objects': 4522, 'num_object_clones': 0, 'num_object_copies': 18088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4522, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4522, 'num_write_kb': 36176, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4559", 'reported_seq': 4622, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:42.637387+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:42.637387+0000', 'last_peered': '2024-09-15T21:05:42.637387+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:42.637387+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4559, 'log_dups_size': 0, 'ondisk_log_size': 4559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37311756, 'num_objects': 4555, 'num_object_clones': 0, 'num_object_copies': 18220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4557, 'num_write_kb': 36452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4408", 'reported_seq': 4482, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:41.181574+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:41.181574+0000', 'last_peered': '2024-09-15T21:05:41.181574+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:41.181574+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4408, 'log_dups_size': 0, 'ondisk_log_size': 4408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36084358, 'num_objects': 4405, 'num_object_clones': 0, 'num_object_copies': 17620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4407, 'num_write_kb': 35246, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2147}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:44.635 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4543", 'reported_seq': 4607, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:42.753500+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:42.753500+0000', 'last_peered': '2024-09-15T21:05:42.753500+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:42.753500+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4543, 'log_dups_size': 0, 'ondisk_log_size': 4543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37180684, 'num_objects': 4539, 'num_object_clones': 0, 'num_object_copies': 18156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4541, 'num_write_kb': 36317, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4637", 'reported_seq': 4688, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.490597+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:43.490597+0000', 'last_peered': '2024-09-15T21:05:43.490597+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:43.490597+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37958924, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 18536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4635, 'num_write_kb': 37077, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2323}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4527", 'reported_seq': 4566, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.375449+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:43.375449+0000', 'last_peered': '2024-09-15T21:05:43.375449+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:43.375449+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4527, 'log_dups_size': 0, 'ondisk_log_size': 4527, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37085184, 'num_objects': 4527, 'num_object_clones': 0, 'num_object_copies': 18108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4527, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4527, 'num_write_kb': 36216, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2211}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4630", 'reported_seq': 4695, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:42.751204+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:42.751204+0000', 'last_peered': '2024-09-15T21:05:42.751204+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:42.751204+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4630, 'log_dups_size': 0, 'ondisk_log_size': 4630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37877004, 'num_objects': 4624, 'num_object_clones': 0, 'num_object_copies': 18496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4628, 'num_write_kb': 37004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2258}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4658", 'reported_seq': 4722, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:42.795858+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:42.795858+0000', 'last_peered': '2024-09-15T21:05:42.795858+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:42.795858+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4658, 'log_dups_size': 0, 'ondisk_log_size': 4658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38114572, 'num_objects': 4653, 'num_object_clones': 0, 'num_object_copies': 18612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4656, 'num_write_kb': 37236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2179}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:44.636 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4541", 'reported_seq': 4596, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.370014+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:43.370014+0000', 'last_peered': '2024-09-15T21:05:43.370014+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:43.370014+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4541, 'log_dups_size': 0, 'ondisk_log_size': 4541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37128728, 'num_objects': 4533, 'num_object_clones': 0, 'num_object_copies': 18132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4538, 'num_write_kb': 36281, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2201}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4493", 'reported_seq': 4569, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:41.320574+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:41.320574+0000', 'last_peered': '2024-09-15T21:05:41.320574+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:41.320574+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4493, 'log_dups_size': 0, 'ondisk_log_size': 4493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36787468, 'num_objects': 4491, 'num_object_clones': 0, 'num_object_copies': 17964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4491, 'num_write_kb': 35926, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2156}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4514", 'reported_seq': 7009, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.537050+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:43.537050+0000', 'last_peered': '2024-09-15T21:05:43.537050+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:43.537050+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4514, 'log_dups_size': 0, 'ondisk_log_size': 4514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36933522, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1040, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4511, 'num_write_kb': 36076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1223, 'num_bytes_recovered': 9994240, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3469}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4469", 'reported_seq': 4502, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.445216+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:43.445216+0000', 'last_peered': '2024-09-15T21:05:43.445216+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:43.445216+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4469, 'log_dups_size': 0, 'ondisk_log_size': 4469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36600454, 'num_objects': 4468, 'num_object_clones': 0, 'num_object_copies': 17872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4468, 'num_write_kb': 35743, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4493", 'reported_seq': 4568, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:41.262454+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:41.262454+0000', 'last_peered': '2024-09-15T21:05:41.262454+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:41.262454+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4493, 'log_dups_size': 0, 'ondisk_log_size': 4493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36797062, 'num_objects': 4492, 'num_object_clones': 0, 'num_object_copies': 17968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4492, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4492, 'num_write_kb': 35935, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4561", 'reported_seq': 4613, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:43.350351+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:43.350351+0000', 'last_peered': '2024-09-15T21:05:43.350351+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:43.350351+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4561, 'log_dups_size': 0, 'ondisk_log_size': 4561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37328140, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 18228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4559, 'num_write_kb': 36468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:44.637 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4500", 'reported_seq': 4578, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:41.342917+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:41.342917+0000', 'last_peered': '2024-09-15T21:05:41.342917+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:41.342917+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4500, 'log_dups_size': 0, 'ondisk_log_size': 4500, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36836620, 'num_objects': 4497, 'num_object_clones': 0, 'num_object_copies': 17988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4497, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4498, 'num_write_kb': 35981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2216}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:44.638 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:44.638 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4562", 'reported_seq': 4626, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:42.789646+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:42.789646+0000', 'last_peered': '2024-09-15T21:05:42.789646+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:42.789646+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4562, 'log_dups_size': 0, 'ondisk_log_size': 4562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37328140, 'num_objects': 4557, 'num_object_clones': 0, 'num_object_copies': 18228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4560, 'num_write_kb': 36468, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:44.638 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:44.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 141 20 73102 73082 4.04871 2.23438 0.0505911 0.0385784 2024-09-15T21:05:45.355 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:45.355 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35863/290252 objects degraded (12.356%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":211,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72563,"data_bytes":594396840,"bytes_used":1920040960,"bytes_avail":384627015680,"bytes_total":386547056640,"degraded_objects":35863,"degraded_total":290252,"degraded_ratio":0.12355814947011562,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":120785,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"write_bytes_sec":2881107,"read_op_per_sec":0,"write_op_per_sec":351},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:45.380 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:45.380 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:45.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 142 20 73364 73344 4.03461 2.04688 0.0510362 0.0387118 2024-09-15T21:05:45.958 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:45.958 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:45.982 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-15T21:05:43.955303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594396840,"num_objects":72563,"num_object_clones":0,"num_object_copies":290252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72590,"num_write_kb":580602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72617,"ondisk_log_size":72617,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1875040,"kb_used_data":1014496,"kb_used_omap":14,"kb_used_meta":860465,"kb_avail":375612320,"statfs":{"total":386547056640,"available":384627015680,"internally_reserved":0,"allocated":1038843904,"data_stored":1038216104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":881116722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,16,3,7,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":34586624,"num_objects":4222,"num_object_clones":0,"num_object_copies":16888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":33776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":1449984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004629"},"pg_stats":[{"pgid":"1.f","version":"31'4522","reported_seq":4562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.518536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:43.518536+0000","last_peered":"2024-09-15T21:05:43.518536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:43.518536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4559","reported_seq":4622,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.637387+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:42.637387+0000","last_peered":"2024-09-15T21:05:42.637387+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:42.637387+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37311756,"num_objects":4555,"num_object_clones":0,"num_object_copies":18220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4557,"num_write_kb":36452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4408","reported_seq":4482,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.181574+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:41.181574+0000","last_peered":"2024-09-15T21:05:41.181574+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:41.181574+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4408,"log_dups_size":0,"ondisk_log_size":4408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36084358,"num_objects":4405,"num_object_clones":0,"num_object_copies":17620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":35246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2147}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4543","reported_seq":4607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.753500+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:42.753500+0000","last_peered":"2024-09-15T21:05:42.753500+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:42.753500+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4543,"log_dups_size":0,"ondisk_log_size":4543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37180684,"num_objects":4539,"num_object_clones":0,"num_object_copies":18156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4541,"num_write_kb":36317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4637","reported_seq":4688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.490597+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:43.490597+0000","last_peered":"2024-09-15T21:05:43.490597+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:43.490597+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958924,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":37077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4527","reported_seq":4566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.375449+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:43.375449+0000","last_peered":"2024-09-15T21:05:43.375449+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:43.375449+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37085184,"num_objects":4527,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4630","reported_seq":4695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.751204+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:42.751204+0000","last_peered":"2024-09-15T21:05:42.751204+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:42.751204+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4630,"log_dups_size":0,"ondisk_log_size":4630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37877004,"num_objects":4624,"num_object_clones":0,"num_object_copies":18496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4628,"num_write_kb":37004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2258}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4658","reported_seq":4722,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.795858+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:42.795858+0000","last_peered":"2024-09-15T21:05:42.795858+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:42.795858+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4658,"log_dups_size":0,"ondisk_log_size":4658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38114572,"num_objects":4653,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2179}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4541","reported_seq":4596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.370014+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:43.370014+0000","last_peered":"2024-09-15T21:05:43.370014+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:43.370014+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37128728,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4493","reported_seq":4569,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.320574+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:41.320574+0000","last_peered":"2024-09-15T21:05:41.320574+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:41.320574+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36787468,"num_objects":4491,"num_object_clones":0,"num_object_copies":17964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4491,"num_write_kb":35926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2156}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4514","reported_seq":7009,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.537050+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:43.537050+0000","last_peered":"2024-09-15T21:05:43.537050+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:43.537050+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36933522,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4469","reported_seq":4502,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.445216+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:43.445216+0000","last_peered":"2024-09-15T21:05:43.445216+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:43.445216+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36600454,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":35743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2212}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4493","reported_seq":4568,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.262454+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:41.262454+0000","last_peered":"2024-09-15T21:05:41.262454+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:41.262454+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36797062,"num_objects":4492,"num_object_clones":0,"num_object_copies":17968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":35935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4561","reported_seq":4613,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.350351+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:43.350351+0000","last_peered":"2024-09-15T21:05:43.350351+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:43.350351+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328140,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4500","reported_seq":4578,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:41.342917+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:41.342917+0000","last_peered":"2024-09-15T21:05:41.342917+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:41.342917+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36836620,"num_objects":4497,"num_object_clones":0,"num_object_copies":17988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4498,"num_write_kb":35981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2216}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4562","reported_seq":4626,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:42.789646+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:42.789646+0000","last_peered":"2024-09-15T21:05:42.789646+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:42.789646+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328140,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4560,"num_write_kb":36468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":594396840,"num_objects":72563,"num_object_clones":0,"num_object_copies":290252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72590,"num_write_kb":580602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1039986688,"data_stored":1039986688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72617,"ondisk_log_size":72617,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521468,"kb_used_data":291244,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93850372,"statfs":{"total":96636764160,"available":96102780928,"internally_reserved":0,"allocated":298233856,"data_stored":298076906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,1,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":517800,"kb_used_data":288088,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":93854040,"statfs":{"total":96636764160,"available":96106536960,"internally_reserved":0,"allocated":295002112,"data_stored":294845162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051606,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307236,"kb_used_data":145108,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94064604,"statfs":{"total":96636764160,"available":96322154496,"internally_reserved":0,"allocated":148590592,"data_stored":148433642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":528536,"kb_used_data":290056,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93843304,"statfs":{"total":96636764160,"available":96095543296,"internally_reserved":0,"allocated":297017344,"data_stored":296860394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":244182734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":297287680,"data_stored":297287680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":149118976,"data_stored":149118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":295436288,"data_stored":295436288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298143744,"data_stored":298143744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:46.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 143 20 73671 73651 4.02317 2.39844 0.0448288 0.0388236 2024-09-15T21:05:47.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 144 20 73981 73961 4.01205 2.42188 0.0417092 0.0389287 2024-09-15T21:05:48.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 145 20 74297 74277 4.00141 2.46875 0.0543627 0.0390356 2024-09-15T21:05:48.984 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:49.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:49.578 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:49.599 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-15T21:05:47.957859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601253544,"num_objects":73400,"num_object_clones":0,"num_object_copies":293600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73427,"num_write_kb":587298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73454,"ondisk_log_size":73454,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1924796,"kb_used_data":1032316,"kb_used_omap":14,"kb_used_meta":892401,"kb_avail":375562564,"statfs":{"total":386547056640,"available":384576065536,"internally_reserved":0,"allocated":1057091584,"data_stored":1056463784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":913819186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,3,14,12,6,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31817728,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3884,"num_write_kb":31072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":974848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005928"},"pg_stats":[{"pgid":"1.f","version":"31'4522","reported_seq":4562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.518536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:43.518536+0000","last_peered":"2024-09-15T21:05:43.518536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:43.518536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37044224,"num_objects":4522,"num_object_clones":0,"num_object_copies":18088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":36176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4651","reported_seq":4714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.754282+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:47.754282+0000","last_peered":"2024-09-15T21:05:47.754282+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:47.754282+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065420,"num_objects":4647,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4571,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.243615+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:46.243615+0000","last_peered":"2024-09-15T21:05:46.243615+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:46.243615+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813446,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4634","reported_seq":4698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.785920+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:47.785920+0000","last_peered":"2024-09-15T21:05:47.785920+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:47.785920+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926156,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":37045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4637","reported_seq":4688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.490597+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:43.490597+0000","last_peered":"2024-09-15T21:05:43.490597+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:43.490597+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958924,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4635,"num_write_kb":37077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2323}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4527","reported_seq":4566,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.375449+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:43.375449+0000","last_peered":"2024-09-15T21:05:43.375449+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:43.375449+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37085184,"num_objects":4527,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":36216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4733","reported_seq":4798,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.773750+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:47.773750+0000","last_peered":"2024-09-15T21:05:47.773750+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:47.773750+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720780,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4731,"num_write_kb":37828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4751","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.649039+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:47.649039+0000","last_peered":"2024-09-15T21:05:47.649039+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:47.649039+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38876428,"num_objects":4746,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4541","reported_seq":4596,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.370014+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:43.370014+0000","last_peered":"2024-09-15T21:05:43.370014+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:43.370014+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4541,"log_dups_size":0,"ondisk_log_size":4541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37128728,"num_objects":4533,"num_object_clones":0,"num_object_copies":18132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4538,"num_write_kb":36281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2201}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4588","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.388367+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:46.388367+0000","last_peered":"2024-09-15T21:05:46.388367+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:46.388367+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565708,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4514","reported_seq":7009,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.537050+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:43.537050+0000","last_peered":"2024-09-15T21:05:43.537050+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:43.537050+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36933522,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4511,"num_write_kb":36076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1223,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3469}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4469","reported_seq":4502,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.445216+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:43.445216+0000","last_peered":"2024-09-15T21:05:43.445216+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:43.445216+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36600454,"num_objects":4468,"num_object_clones":0,"num_object_copies":17872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":35743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2212}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4587","reported_seq":4662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.283950+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:46.283950+0000","last_peered":"2024-09-15T21:05:46.283950+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:46.283950+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4561","reported_seq":4613,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:43.350351+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:43.350351+0000","last_peered":"2024-09-15T21:05:43.350351+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:43.350351+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37328140,"num_objects":4557,"num_object_clones":0,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":36468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2278}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4592","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.375283+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:46.375283+0000","last_peered":"2024-09-15T21:05:46.375283+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:46.375283+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590284,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4650","reported_seq":4713,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.746429+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:47.746429+0000","last_peered":"2024-09-15T21:05:47.746429+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:47.746429+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049036,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":601253544,"num_objects":73400,"num_object_clones":0,"num_object_copies":293600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73427,"num_write_kb":587298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":9994240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1058222080,"data_stored":1058222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73454,"ondisk_log_size":73454,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":521468,"kb_used_data":291244,"kb_used_omap":4,"kb_used_meta":230203,"kb_avail":93850372,"statfs":{"total":96636764160,"available":96102780928,"internally_reserved":0,"allocated":298233856,"data_stored":298076906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":235728592},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,1,4,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545916,"kb_used_data":293996,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93825924,"statfs":{"total":96636764160,"available":96077746176,"internally_reserved":0,"allocated":301051904,"data_stored":300894954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317416,"kb_used_data":151128,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94054424,"statfs":{"total":96636764160,"available":96311730176,"internally_reserved":0,"allocated":154755072,"data_stored":154598122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539996,"kb_used_data":295948,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93831844,"statfs":{"total":96636764160,"available":96083808256,"internally_reserved":0,"allocated":303050752,"data_stored":302893802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303271936,"data_stored":303271936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155369472,"data_stored":155369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301436928,"data_stored":301436928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":298143744,"data_stored":298143744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:49.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:49.647 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 146 20 74587 74567 3.98952 2.26562 0.0595717 0.0391494 2024-09-15T21:05:50.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:50.190 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:50.216 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-15T21:05:49.958505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606742184,"num_objects":74070,"num_object_clones":0,"num_object_copies":296280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74097,"num_write_kb":592658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74124,"ondisk_log_size":74124,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1952780,"kb_used_data":1038088,"kb_used_omap":14,"kb_used_meta":914609,"kb_avail":375534580,"statfs":{"total":386547056640,"available":384547409920,"internally_reserved":0,"allocated":1063002112,"data_stored":1062374312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":936560178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,6,7,15,8,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29999104,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":29296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005880"},"pg_stats":[{"pgid":"1.f","version":"31'4621","reported_seq":4662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.507548+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:48.507548+0000","last_peered":"2024-09-15T21:05:48.507548+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:48.507548+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37855232,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4651","reported_seq":4714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.754282+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:47.754282+0000","last_peered":"2024-09-15T21:05:47.754282+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:47.754282+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065420,"num_objects":4647,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4571,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.243615+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:46.243615+0000","last_peered":"2024-09-15T21:05:46.243615+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:46.243615+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813446,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4634","reported_seq":4698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.785920+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:47.785920+0000","last_peered":"2024-09-15T21:05:47.785920+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:47.785920+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926156,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":37045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4727","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.576958+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:48.576958+0000","last_peered":"2024-09-15T21:05:48.576958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:48.576958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38696204,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4725,"num_write_kb":37797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4616","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.502604+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:48.502604+0000","last_peered":"2024-09-15T21:05:48.502604+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:48.502604+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4616,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4733","reported_seq":4798,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.773750+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:47.773750+0000","last_peered":"2024-09-15T21:05:47.773750+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:47.773750+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720780,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4731,"num_write_kb":37828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4751","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.649039+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:47.649039+0000","last_peered":"2024-09-15T21:05:47.649039+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:47.649039+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38876428,"num_objects":4746,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4642","reported_seq":4696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.606984+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:48.606984+0000","last_peered":"2024-09-15T21:05:48.606984+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:48.606984+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37956120,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4588","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.388367+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:46.388367+0000","last_peered":"2024-09-15T21:05:46.388367+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:46.388367+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565708,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4609","reported_seq":7234,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.618478+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:48.618478+0000","last_peered":"2024-09-15T21:05:48.618478+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:48.618478+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37711762,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1288,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4577","reported_seq":4610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.627842+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:48.627842+0000","last_peered":"2024-09-15T21:05:48.627842+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:48.627842+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37485190,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":36607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4587","reported_seq":4662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.283950+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:46.283950+0000","last_peered":"2024-09-15T21:05:46.283950+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:46.283950+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4649","reported_seq":4701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.571524+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:48.571524+0000","last_peered":"2024-09-15T21:05:48.571524+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:48.571524+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049036,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4647,"num_write_kb":37172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2366}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4592","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.375283+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:46.375283+0000","last_peered":"2024-09-15T21:05:46.375283+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:46.375283+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590284,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4650","reported_seq":4713,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.746429+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:47.746429+0000","last_peered":"2024-09-15T21:05:47.746429+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:47.746429+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049036,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606742184,"num_objects":74070,"num_object_clones":0,"num_object_copies":296280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74097,"num_write_kb":592658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064370176,"data_stored":1064370176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74124,"ondisk_log_size":74124,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549452,"kb_used_data":297016,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93822388,"statfs":{"total":96636764160,"available":96074125312,"internally_reserved":0,"allocated":304144384,"data_stored":303987434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545916,"kb_used_data":293996,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93825924,"statfs":{"total":96636764160,"available":96077746176,"internally_reserved":0,"allocated":301051904,"data_stored":300894954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317416,"kb_used_data":151128,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94054424,"statfs":{"total":96636764160,"available":96311730176,"internally_reserved":0,"allocated":154755072,"data_stored":154598122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539996,"kb_used_data":295948,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93831844,"statfs":{"total":96636764160,"available":96083808256,"internally_reserved":0,"allocated":303050752,"data_stored":302893802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303271936,"data_stored":303271936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155369472,"data_stored":155369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301436928,"data_stored":301436928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304291840,"data_stored":304291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:50.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4621", 'reported_seq': 4662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.507548+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:48.507548+0000', 'last_peered': '2024-09-15T21:05:48.507548+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:48.507548+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4621, 'log_dups_size': 0, 'ondisk_log_size': 4621, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37855232, 'num_objects': 4621, 'num_object_clones': 0, 'num_object_copies': 18484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4621, 'num_write_kb': 36968, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4651", 'reported_seq': 4714, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:47.754282+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:47.754282+0000', 'last_peered': '2024-09-15T21:05:47.754282+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:47.754282+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4651, 'log_dups_size': 0, 'ondisk_log_size': 4651, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38065420, 'num_objects': 4647, 'num_object_clones': 0, 'num_object_copies': 18588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4649, 'num_write_kb': 37188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4497", 'reported_seq': 4571, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:46.243615+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:46.243615+0000', 'last_peered': '2024-09-15T21:05:46.243615+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:46.243615+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4497, 'log_dups_size': 0, 'ondisk_log_size': 4497, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36813446, 'num_objects': 4494, 'num_object_clones': 0, 'num_object_copies': 17976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4496, 'num_write_kb': 35958, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2236}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4634", 'reported_seq': 4698, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:47.785920+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:47.785920+0000', 'last_peered': '2024-09-15T21:05:47.785920+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:47.785920+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4634, 'log_dups_size': 0, 'ondisk_log_size': 4634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37926156, 'num_objects': 4630, 'num_object_clones': 0, 'num_object_copies': 18520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4632, 'num_write_kb': 37045, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2296}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:50.218 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4727", 'reported_seq': 4778, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.576958+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:48.576958+0000', 'last_peered': '2024-09-15T21:05:48.576958+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:48.576958+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4727, 'log_dups_size': 0, 'ondisk_log_size': 4727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38696204, 'num_objects': 4724, 'num_object_clones': 0, 'num_object_copies': 18896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4724, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4725, 'num_write_kb': 37797, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2413}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4616", 'reported_seq': 4656, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.502604+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:48.502604+0000', 'last_peered': '2024-09-15T21:05:48.502604+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:48.502604+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4616, 'log_dups_size': 0, 'ondisk_log_size': 4616, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37814272, 'num_objects': 4616, 'num_object_clones': 0, 'num_object_copies': 18464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4616, 'num_write_kb': 36928, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2300}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4733", 'reported_seq': 4798, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:47.773750+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:47.773750+0000', 'last_peered': '2024-09-15T21:05:47.773750+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:47.773750+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4733, 'log_dups_size': 0, 'ondisk_log_size': 4733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38720780, 'num_objects': 4727, 'num_object_clones': 0, 'num_object_copies': 18908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4731, 'num_write_kb': 37828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4751", 'reported_seq': 4815, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:47.649039+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:47.649039+0000', 'last_peered': '2024-09-15T21:05:47.649039+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:47.649039+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4751, 'log_dups_size': 0, 'ondisk_log_size': 4751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38876428, 'num_objects': 4746, 'num_object_clones': 0, 'num_object_copies': 18984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4749, 'num_write_kb': 37980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4642", 'reported_seq': 4696, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.606984+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:48.606984+0000', 'last_peered': '2024-09-15T21:05:48.606984+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:48.606984+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4642, 'log_dups_size': 0, 'ondisk_log_size': 4642, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37956120, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 18536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4639, 'num_write_kb': 37089, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2302}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4588", 'reported_seq': 4663, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:46.388367+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:46.388367+0000', 'last_peered': '2024-09-15T21:05:46.388367+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:46.388367+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4588, 'log_dups_size': 0, 'ondisk_log_size': 4588, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37565708, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4586, 'num_write_kb': 36686, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2251}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:50.219 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4609", 'reported_seq': 7234, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.618478+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:48.618478+0000', 'last_peered': '2024-09-15T21:05:48.618478+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:48.618478+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4609, 'log_dups_size': 0, 'ondisk_log_size': 4609, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37711762, 'num_objects': 4604, 'num_object_clones': 0, 'num_object_copies': 18416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 978, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4606, 'num_write_kb': 36836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1288, 'num_bytes_recovered': 10502144, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3626}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4577", 'reported_seq': 4610, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.627842+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:48.627842+0000', 'last_peered': '2024-09-15T21:05:48.627842+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:48.627842+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4577, 'log_dups_size': 0, 'ondisk_log_size': 4577, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37485190, 'num_objects': 4576, 'num_object_clones': 0, 'num_object_copies': 18304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4576, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4576, 'num_write_kb': 36607, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2320}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4587", 'reported_seq': 4662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:46.283950+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:46.283950+0000', 'last_peered': '2024-09-15T21:05:46.283950+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:46.283950+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4587, 'log_dups_size': 0, 'ondisk_log_size': 4587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37567110, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4586, 'num_write_kb': 36687, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2300}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4649", 'reported_seq': 4701, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:48.571524+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:48.571524+0000', 'last_peered': '2024-09-15T21:05:48.571524+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:48.571524+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4649, 'log_dups_size': 0, 'ondisk_log_size': 4649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38049036, 'num_objects': 4645, 'num_object_clones': 0, 'num_object_copies': 18580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4647, 'num_write_kb': 37172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2366}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4592", 'reported_seq': 4668, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:46.375283+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:46.375283+0000', 'last_peered': '2024-09-15T21:05:46.375283+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:46.375283+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4592, 'log_dups_size': 0, 'ondisk_log_size': 4592, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37590284, 'num_objects': 4589, 'num_object_clones': 0, 'num_object_copies': 18356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4590, 'num_write_kb': 36717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:50.220 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:50.221 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4650", 'reported_seq': 4713, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:47.746429+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:47.746429+0000', 'last_peered': '2024-09-15T21:05:47.746429+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:47.746429+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4650, 'log_dups_size': 0, 'ondisk_log_size': 4650, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38049036, 'num_objects': 4645, 'num_object_clones': 0, 'num_object_copies': 18580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4645, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4648, 'num_write_kb': 37172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:50.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:50.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 147 20 74859 74839 3.97683 2.125 0.050541 0.0392736 2024-09-15T21:05:50.979 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:50.979 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35801/296280 objects degraded (12.084%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":216,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74070,"data_bytes":606742184,"bytes_used":1999646720,"bytes_avail":384547409920,"bytes_total":386547056640,"degraded_objects":35801,"degraded_total":296280,"degraded_ratio":0.12083502092615094,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":123502,"recovering_keys_per_sec":0,"num_objects_recovered":184,"num_bytes_recovered":1482752,"num_keys_recovered":0,"write_bytes_sec":2498700,"read_op_per_sec":0,"write_op_per_sec":305},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:51.025 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:51.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:51.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 148 20 75135 75115 3.96453 2.15625 0.0564847 0.0393977 2024-09-15T21:05:51.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:51.661 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:51.683 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-09-15T21:05:49.958505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":606742184,"num_objects":74070,"num_object_clones":0,"num_object_copies":296280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74097,"num_write_kb":592658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74124,"ondisk_log_size":74124,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1952780,"kb_used_data":1038088,"kb_used_omap":14,"kb_used_meta":914609,"kb_avail":375534580,"statfs":{"total":386547056640,"available":384547409920,"internally_reserved":0,"allocated":1063002112,"data_stored":1062374312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":936560178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,6,7,15,8,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29999104,"num_objects":3662,"num_object_clones":0,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3662,"num_write_kb":29296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005880"},"pg_stats":[{"pgid":"1.f","version":"31'4621","reported_seq":4662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.507548+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:48.507548+0000","last_peered":"2024-09-15T21:05:48.507548+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:48.507548+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37855232,"num_objects":4621,"num_object_clones":0,"num_object_copies":18484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":36968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4651","reported_seq":4714,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.754282+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:47.754282+0000","last_peered":"2024-09-15T21:05:47.754282+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:47.754282+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38065420,"num_objects":4647,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":37188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2267}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4497","reported_seq":4571,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.243615+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:46.243615+0000","last_peered":"2024-09-15T21:05:46.243615+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:46.243615+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4497,"log_dups_size":0,"ondisk_log_size":4497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36813446,"num_objects":4494,"num_object_clones":0,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4496,"num_write_kb":35958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2236}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4634","reported_seq":4698,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.785920+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:47.785920+0000","last_peered":"2024-09-15T21:05:47.785920+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:47.785920+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37926156,"num_objects":4630,"num_object_clones":0,"num_object_copies":18520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":37045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2296}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4727","reported_seq":4778,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.576958+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:48.576958+0000","last_peered":"2024-09-15T21:05:48.576958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:48.576958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4727,"log_dups_size":0,"ondisk_log_size":4727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38696204,"num_objects":4724,"num_object_clones":0,"num_object_copies":18896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4725,"num_write_kb":37797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2413}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4616","reported_seq":4656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.502604+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:48.502604+0000","last_peered":"2024-09-15T21:05:48.502604+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:48.502604+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":4616,"num_object_clones":0,"num_object_copies":18464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4616,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2300}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4733","reported_seq":4798,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.773750+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:47.773750+0000","last_peered":"2024-09-15T21:05:47.773750+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:47.773750+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4733,"log_dups_size":0,"ondisk_log_size":4733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38720780,"num_objects":4727,"num_object_clones":0,"num_object_copies":18908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4731,"num_write_kb":37828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2361}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4751","reported_seq":4815,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.649039+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:47.649039+0000","last_peered":"2024-09-15T21:05:47.649039+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:47.649039+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38876428,"num_objects":4746,"num_object_clones":0,"num_object_copies":18984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4749,"num_write_kb":37980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2272}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4642","reported_seq":4696,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.606984+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:48.606984+0000","last_peered":"2024-09-15T21:05:48.606984+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:48.606984+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37956120,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4639,"num_write_kb":37089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2302}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4588","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.388367+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:46.388367+0000","last_peered":"2024-09-15T21:05:46.388367+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:46.388367+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37565708,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2251}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4609","reported_seq":7234,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.618478+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:48.618478+0000","last_peered":"2024-09-15T21:05:48.618478+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:48.618478+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4609,"log_dups_size":0,"ondisk_log_size":4609,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37711762,"num_objects":4604,"num_object_clones":0,"num_object_copies":18416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":36836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1288,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3626}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4577","reported_seq":4610,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.627842+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:48.627842+0000","last_peered":"2024-09-15T21:05:48.627842+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:48.627842+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37485190,"num_objects":4576,"num_object_clones":0,"num_object_copies":18304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":36607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2320}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4587","reported_seq":4662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.283950+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:46.283950+0000","last_peered":"2024-09-15T21:05:46.283950+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:46.283950+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4586,"num_write_kb":36687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2300}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4649","reported_seq":4701,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:48.571524+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:48.571524+0000","last_peered":"2024-09-15T21:05:48.571524+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:48.571524+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049036,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4647,"num_write_kb":37172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2366}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4592","reported_seq":4668,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:46.375283+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:46.375283+0000","last_peered":"2024-09-15T21:05:46.375283+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:46.375283+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37590284,"num_objects":4589,"num_object_clones":0,"num_object_copies":18356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4590,"num_write_kb":36717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2308}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4650","reported_seq":4713,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:47.746429+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:47.746429+0000","last_peered":"2024-09-15T21:05:47.746429+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:47.746429+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38049036,"num_objects":4645,"num_object_clones":0,"num_object_copies":18580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":37172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":606742184,"num_objects":74070,"num_object_clones":0,"num_object_copies":296280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74097,"num_write_kb":592658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1312,"num_bytes_recovered":10502144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064370176,"data_stored":1064370176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74124,"ondisk_log_size":74124,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549452,"kb_used_data":297016,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93822388,"statfs":{"total":96636764160,"available":96074125312,"internally_reserved":0,"allocated":304144384,"data_stored":303987434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,4,6,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545916,"kb_used_data":293996,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93825924,"statfs":{"total":96636764160,"available":96077746176,"internally_reserved":0,"allocated":301051904,"data_stored":300894954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051607,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":317416,"kb_used_data":151128,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94054424,"statfs":{"total":96636764160,"available":96311730176,"internally_reserved":0,"allocated":154755072,"data_stored":154598122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,6,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":539996,"kb_used_data":295948,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93831844,"statfs":{"total":96636764160,"available":96083808256,"internally_reserved":0,"allocated":303050752,"data_stored":302893802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,5,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303271936,"data_stored":303271936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":155369472,"data_stored":155369472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":301436928,"data_stored":301436928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":304291840,"data_stored":304291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:52.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 149 20 75441 75421 3.95397 2.39062 0.041941 0.0395022 2024-09-15T21:05:53.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 150 20 75763 75743 3.94438 2.51562 0.0529689 0.0396011 2024-09-15T21:05:54.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 151 20 76074 76054 3.93434 2.42969 0.0346489 0.0396996 2024-09-15T21:05:54.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:55.299 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:55.299 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:55.320 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-15T21:05:53.960099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1980816,"kb_used_data":1061968,"kb_used_omap":14,"kb_used_meta":918769,"kb_avail":375506544,"statfs":{"total":386547056640,"available":384518701056,"internally_reserved":0,"allocated":1087455232,"data_stored":1086827432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":940820018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,14,15,19,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33439744,"num_objects":4082,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":32656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005674"},"pg_stats":[{"pgid":"1.f","version":"31'4721","reported_seq":4761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.626678+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:53.626678+0000","last_peered":"2024-09-15T21:05:53.626678+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:53.626678+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38674432,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":37768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.817784+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:52.817784+0000","last_peered":"2024-09-15T21:05:52.817784+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:52.817784+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4589","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.396058+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:51.396058+0000","last_peered":"2024-09-15T21:05:51.396058+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:51.396058+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4724","reported_seq":4788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.900872+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:52.900872+0000","last_peered":"2024-09-15T21:05:52.900872+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:52.900872+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663436,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4820","reported_seq":4871,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.594453+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:53.594453+0000","last_peered":"2024-09-15T21:05:53.594453+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:53.594453+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4707","reported_seq":4747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.590972+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:53.590972+0000","last_peered":"2024-09-15T21:05:53.590972+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:53.590972+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38559744,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4831","reported_seq":4896,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.877666+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:52.877666+0000","last_peered":"2024-09-15T21:05:52.877666+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:52.877666+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39523596,"num_objects":4825,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4834","reported_seq":4898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.848944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:52.848944+0000","last_peered":"2024-09-15T21:05:52.848944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:52.848944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4832,"num_write_kb":38644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4736","reported_seq":4791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.476215+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:53.476215+0000","last_peered":"2024-09-15T21:05:53.476215+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:53.476215+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38726168,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4679","reported_seq":4753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.373117+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:51.373117+0000","last_peered":"2024-09-15T21:05:51.373117+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:51.373117+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311180,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4698","reported_seq":7444,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.589473+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:53.589473+0000","last_peered":"2024-09-15T21:05:53.589473+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:53.589473+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38440850,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4657","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.545983+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:53.545983+0000","last_peered":"2024-09-15T21:05:53.545983+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:53.545983+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38140550,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4676","reported_seq":4751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.338221+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:51.338221+0000","last_peered":"2024-09-15T21:05:51.338221+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:51.338221+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296198,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4732","reported_seq":4784,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.473223+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:53.473223+0000","last_peered":"2024-09-15T21:05:53.473223+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:53.473223+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728972,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4692","reported_seq":4770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.375207+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:51.375207+0000","last_peered":"2024-09-15T21:05:51.375207+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:51.375207+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38409484,"num_objects":4689,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.792635+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:52.792635+0000","last_peered":"2024-09-15T21:05:52.792635+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:52.792635+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088876544,"data_stored":1088876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559488,"kb_used_data":302896,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93812352,"statfs":{"total":96636764160,"available":96063848448,"internally_reserved":0,"allocated":310165504,"data_stored":310008554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,1,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556024,"kb_used_data":299944,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93815816,"statfs":{"total":96636764160,"available":96067395584,"internally_reserved":0,"allocated":307142656,"data_stored":306985706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,3,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315236,"kb_used_data":157268,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94056604,"statfs":{"total":96636764160,"available":96313962496,"internally_reserved":0,"allocated":161042432,"data_stored":160885482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550068,"kb_used_data":301860,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93821772,"statfs":{"total":96636764160,"available":96073494528,"internally_reserved":0,"allocated":309104640,"data_stored":308947690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309379072,"data_stored":309379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161689600,"data_stored":161689600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307478528,"data_stored":307478528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:55.321 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:55.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 152 20 76354 76334 3.92285 2.1875 0.0466887 0.0398137 2024-09-15T21:05:55.899 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:55.899 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:55.927 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-09-15T21:05:53.960099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1980816,"kb_used_data":1061968,"kb_used_omap":14,"kb_used_meta":918769,"kb_avail":375506544,"statfs":{"total":386547056640,"available":384518701056,"internally_reserved":0,"allocated":1087455232,"data_stored":1086827432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":940820018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,14,15,19,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":33439744,"num_objects":4082,"num_object_clones":0,"num_object_copies":16328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4082,"num_write_kb":32656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":1482752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005674"},"pg_stats":[{"pgid":"1.f","version":"31'4721","reported_seq":4761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.626678+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:53.626678+0000","last_peered":"2024-09-15T21:05:53.626678+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:53.626678+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38674432,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":37768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.817784+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:52.817784+0000","last_peered":"2024-09-15T21:05:52.817784+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:52.817784+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4589","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.396058+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:51.396058+0000","last_peered":"2024-09-15T21:05:51.396058+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:51.396058+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4724","reported_seq":4788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.900872+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:52.900872+0000","last_peered":"2024-09-15T21:05:52.900872+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:52.900872+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663436,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4820","reported_seq":4871,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.594453+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:53.594453+0000","last_peered":"2024-09-15T21:05:53.594453+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:53.594453+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4707","reported_seq":4747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.590972+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:53.590972+0000","last_peered":"2024-09-15T21:05:53.590972+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:53.590972+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38559744,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4831","reported_seq":4896,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.877666+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:52.877666+0000","last_peered":"2024-09-15T21:05:52.877666+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:52.877666+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39523596,"num_objects":4825,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4834","reported_seq":4898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.848944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:52.848944+0000","last_peered":"2024-09-15T21:05:52.848944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:52.848944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4832,"num_write_kb":38644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4736","reported_seq":4791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.476215+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:53.476215+0000","last_peered":"2024-09-15T21:05:53.476215+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:53.476215+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38726168,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4679","reported_seq":4753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.373117+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:51.373117+0000","last_peered":"2024-09-15T21:05:51.373117+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:51.373117+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311180,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4698","reported_seq":7444,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.589473+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:53.589473+0000","last_peered":"2024-09-15T21:05:53.589473+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:53.589473+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38440850,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4657","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.545983+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:53.545983+0000","last_peered":"2024-09-15T21:05:53.545983+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:53.545983+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38140550,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4676","reported_seq":4751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.338221+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:51.338221+0000","last_peered":"2024-09-15T21:05:51.338221+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:51.338221+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296198,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4732","reported_seq":4784,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.473223+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:53.473223+0000","last_peered":"2024-09-15T21:05:53.473223+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:53.473223+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728972,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4692","reported_seq":4770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.375207+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:51.375207+0000","last_peered":"2024-09-15T21:05:51.375207+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:51.375207+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38409484,"num_objects":4689,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.792635+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:52.792635+0000","last_peered":"2024-09-15T21:05:52.792635+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:52.792635+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088876544,"data_stored":1088876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559488,"kb_used_data":302896,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93812352,"statfs":{"total":96636764160,"available":96063848448,"internally_reserved":0,"allocated":310165504,"data_stored":310008554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,1,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556024,"kb_used_data":299944,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93815816,"statfs":{"total":96636764160,"available":96067395584,"internally_reserved":0,"allocated":307142656,"data_stored":306985706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,3,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315236,"kb_used_data":157268,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94056604,"statfs":{"total":96636764160,"available":96313962496,"internally_reserved":0,"allocated":161042432,"data_stored":160885482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550068,"kb_used_data":301860,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93821772,"statfs":{"total":96636764160,"available":96073494528,"internally_reserved":0,"allocated":309104640,"data_stored":308947690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309379072,"data_stored":309379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161689600,"data_stored":161689600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307478528,"data_stored":307478528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4721", 'reported_seq': 4761, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.626678+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:53.626678+0000', 'last_peered': '2024-09-15T21:05:53.626678+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:53.626678+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4721, 'log_dups_size': 0, 'ondisk_log_size': 4721, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38674432, 'num_objects': 4721, 'num_object_clones': 0, 'num_object_copies': 18884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4721, 'num_write_kb': 37768, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2445}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4739", 'reported_seq': 4802, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:52.817784+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:52.817784+0000', 'last_peered': '2024-09-15T21:05:52.817784+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:52.817784+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4739, 'log_dups_size': 0, 'ondisk_log_size': 4739, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38786316, 'num_objects': 4735, 'num_object_clones': 0, 'num_object_copies': 18940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4737, 'num_write_kb': 37892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:05:55.928 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4589", 'reported_seq': 4663, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:51.396058+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:51.396058+0000', 'last_peered': '2024-09-15T21:05:51.396058+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:51.396058+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4589, 'log_dups_size': 0, 'ondisk_log_size': 4589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37567110, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4588, 'num_write_kb': 36694, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2328}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4724", 'reported_seq': 4788, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:52.900872+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:52.900872+0000', 'last_peered': '2024-09-15T21:05:52.900872+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:52.900872+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4724, 'log_dups_size': 0, 'ondisk_log_size': 4724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38663436, 'num_objects': 4720, 'num_object_clones': 0, 'num_object_copies': 18880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4722, 'num_write_kb': 37765, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4820", 'reported_seq': 4871, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.594453+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:53.594453+0000', 'last_peered': '2024-09-15T21:05:53.594453+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:53.594453+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4820, 'log_dups_size': 0, 'ondisk_log_size': 4820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39458060, 'num_objects': 4817, 'num_object_clones': 0, 'num_object_copies': 19268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4818, 'num_write_kb': 38541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2506}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4707", 'reported_seq': 4747, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.590972+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:53.590972+0000', 'last_peered': '2024-09-15T21:05:53.590972+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:53.590972+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4707, 'log_dups_size': 0, 'ondisk_log_size': 4707, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38559744, 'num_objects': 4707, 'num_object_clones': 0, 'num_object_copies': 18828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4707, 'num_write_kb': 37656, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2391}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4831", 'reported_seq': 4896, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:52.877666+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:52.877666+0000', 'last_peered': '2024-09-15T21:05:52.877666+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:52.877666+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4831, 'log_dups_size': 0, 'ondisk_log_size': 4831, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39523596, 'num_objects': 4825, 'num_object_clones': 0, 'num_object_copies': 19300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4829, 'num_write_kb': 38612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2459}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:05:55.929 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4834", 'reported_seq': 4898, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:52.848944+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:52.848944+0000', 'last_peered': '2024-09-15T21:05:52.848944+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:52.848944+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4834, 'log_dups_size': 0, 'ondisk_log_size': 4834, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39556364, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 19316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4832, 'num_write_kb': 38644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2355}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4736", 'reported_seq': 4791, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.476215+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:53.476215+0000', 'last_peered': '2024-09-15T21:05:53.476215+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:53.476215+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4736, 'log_dups_size': 0, 'ondisk_log_size': 4736, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38726168, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 18912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2396}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4679", 'reported_seq': 4753, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:51.373117+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:51.373117+0000', 'last_peered': '2024-09-15T21:05:51.373117+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:51.373117+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4679, 'log_dups_size': 0, 'ondisk_log_size': 4679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38311180, 'num_objects': 4677, 'num_object_clones': 0, 'num_object_copies': 18708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4677, 'num_write_kb': 37414, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2342}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4698", 'reported_seq': 7444, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.589473+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:53.589473+0000', 'last_peered': '2024-09-15T21:05:53.589473+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:53.589473+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4698, 'log_dups_size': 0, 'ondisk_log_size': 4698, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38440850, 'num_objects': 4693, 'num_object_clones': 0, 'num_object_copies': 18772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 918, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4695, 'num_write_kb': 37548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1348, 'num_bytes_recovered': 10993664, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3775}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4657", 'reported_seq': 4692, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.545983+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:53.545983+0000', 'last_peered': '2024-09-15T21:05:53.545983+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:53.545983+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4657, 'log_dups_size': 0, 'ondisk_log_size': 4657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38140550, 'num_objects': 4656, 'num_object_clones': 0, 'num_object_copies': 18624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4656, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4656, 'num_write_kb': 37247, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2400}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4676", 'reported_seq': 4751, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:51.338221+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:51.338221+0000', 'last_peered': '2024-09-15T21:05:51.338221+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:51.338221+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4676, 'log_dups_size': 0, 'ondisk_log_size': 4676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38296198, 'num_objects': 4675, 'num_object_clones': 0, 'num_object_copies': 18700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4675, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4675, 'num_write_kb': 37399, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2389}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:05:55.930 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4732", 'reported_seq': 4784, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:53.473223+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:53.473223+0000', 'last_peered': '2024-09-15T21:05:53.473223+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:53.473223+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4732, 'log_dups_size': 0, 'ondisk_log_size': 4732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38728972, 'num_objects': 4728, 'num_object_clones': 0, 'num_object_copies': 18912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4728, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4730, 'num_write_kb': 37836, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2449}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:05:55.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:05:55.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4692", 'reported_seq': 4770, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:51.375207+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:51.375207+0000', 'last_peered': '2024-09-15T21:05:51.375207+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:51.375207+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4692, 'log_dups_size': 0, 'ondisk_log_size': 4692, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38409484, 'num_objects': 4689, 'num_object_clones': 0, 'num_object_copies': 18756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4690, 'num_write_kb': 37517, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:05:55.931 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:05:55.931 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4735", 'reported_seq': 4799, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:52.792635+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:52.792635+0000', 'last_peered': '2024-09-15T21:05:52.792635+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:52.792635+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4735, 'log_dups_size': 0, 'ondisk_log_size': 4735, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38745356, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4733, 'num_write_kb': 37852, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:05:55.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:05:56.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 153 20 76630 76610 3.9113 2.15625 0.0469165 0.0399304 2024-09-15T21:05:56.699 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:56.699 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35741/302064 objects degraded (11.832%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":222,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":75516,"data_bytes":618587816,"bytes_used":2028355584,"bytes_avail":384518701056,"bytes_total":386547056640,"degraded_objects":35741,"degraded_total":302064,"degraded_ratio":0.1183226071296149,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":83242,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":999424,"num_keys_recovered":0,"write_bytes_sec":2014869,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:05:56.727 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:05:56.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:05:57.317 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:05:57.317 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:05:57.346 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-15T21:05:55.961525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1980816,"kb_used_data":1061968,"kb_used_omap":14,"kb_used_meta":918769,"kb_avail":375506544,"statfs":{"total":386547056640,"available":384518701056,"internally_reserved":0,"allocated":1087455232,"data_stored":1086827432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":940820018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,14,15,19,5,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24190976,"num_objects":2953,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":23624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":999424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006222"},"pg_stats":[{"pgid":"1.f","version":"31'4721","reported_seq":4761,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.626678+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:53.626678+0000","last_peered":"2024-09-15T21:05:53.626678+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:53.626678+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4721,"log_dups_size":0,"ondisk_log_size":4721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38674432,"num_objects":4721,"num_object_clones":0,"num_object_copies":18884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4721,"num_write_kb":37768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4739","reported_seq":4802,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.817784+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:52.817784+0000","last_peered":"2024-09-15T21:05:52.817784+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:52.817784+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4589","reported_seq":4663,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.396058+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:51.396058+0000","last_peered":"2024-09-15T21:05:51.396058+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:51.396058+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37567110,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4588,"num_write_kb":36694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2328}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4724","reported_seq":4788,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.900872+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:52.900872+0000","last_peered":"2024-09-15T21:05:52.900872+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:52.900872+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38663436,"num_objects":4720,"num_object_clones":0,"num_object_copies":18880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":37765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4820","reported_seq":4871,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.594453+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:53.594453+0000","last_peered":"2024-09-15T21:05:53.594453+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:53.594453+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39458060,"num_objects":4817,"num_object_clones":0,"num_object_copies":19268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4818,"num_write_kb":38541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2506}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4707","reported_seq":4747,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.590972+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:53.590972+0000","last_peered":"2024-09-15T21:05:53.590972+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:53.590972+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38559744,"num_objects":4707,"num_object_clones":0,"num_object_copies":18828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4707,"num_write_kb":37656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2391}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4831","reported_seq":4896,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.877666+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:52.877666+0000","last_peered":"2024-09-15T21:05:52.877666+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:52.877666+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4831,"log_dups_size":0,"ondisk_log_size":4831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39523596,"num_objects":4825,"num_object_clones":0,"num_object_copies":19300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4829,"num_write_kb":38612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4834","reported_seq":4898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.848944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:52.848944+0000","last_peered":"2024-09-15T21:05:52.848944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:52.848944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4832,"num_write_kb":38644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2355}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4736","reported_seq":4791,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.476215+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:53.476215+0000","last_peered":"2024-09-15T21:05:53.476215+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:53.476215+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38726168,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2396}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4679","reported_seq":4753,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.373117+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:51.373117+0000","last_peered":"2024-09-15T21:05:51.373117+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:51.373117+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38311180,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4677,"num_write_kb":37414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2342}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4698","reported_seq":7444,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.589473+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:53.589473+0000","last_peered":"2024-09-15T21:05:53.589473+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:53.589473+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38440850,"num_objects":4693,"num_object_clones":0,"num_object_copies":18772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4695,"num_write_kb":37548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1348,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3775}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4657","reported_seq":4692,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.545983+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:53.545983+0000","last_peered":"2024-09-15T21:05:53.545983+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:53.545983+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38140550,"num_objects":4656,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4656,"num_write_kb":37247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2400}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4676","reported_seq":4751,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.338221+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:51.338221+0000","last_peered":"2024-09-15T21:05:51.338221+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:51.338221+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38296198,"num_objects":4675,"num_object_clones":0,"num_object_copies":18700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4675,"num_write_kb":37399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2389}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4732","reported_seq":4784,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:53.473223+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:53.473223+0000","last_peered":"2024-09-15T21:05:53.473223+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:53.473223+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38728972,"num_objects":4728,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4730,"num_write_kb":37836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2449}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4692","reported_seq":4770,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:51.375207+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:51.375207+0000","last_peered":"2024-09-15T21:05:51.375207+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:51.375207+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38409484,"num_objects":4689,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4690,"num_write_kb":37517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4735","reported_seq":4799,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:52.792635+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:52.792635+0000","last_peered":"2024-09-15T21:05:52.792635+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:52.792635+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38745356,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4733,"num_write_kb":37852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":618587816,"num_objects":75516,"num_object_clones":0,"num_object_copies":302064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75543,"num_write_kb":604226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1372,"num_bytes_recovered":10993664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088876544,"data_stored":1088876544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75570,"ondisk_log_size":75570,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559488,"kb_used_data":302896,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93812352,"statfs":{"total":96636764160,"available":96063848448,"internally_reserved":0,"allocated":310165504,"data_stored":310008554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,1,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556024,"kb_used_data":299944,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93815816,"statfs":{"total":96636764160,"available":96067395584,"internally_reserved":0,"allocated":307142656,"data_stored":306985706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,9,3,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051608,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315236,"kb_used_data":157268,"kb_used_omap":4,"kb_used_meta":157947,"kb_avail":94056604,"statfs":{"total":96636764160,"available":96313962496,"internally_reserved":0,"allocated":161042432,"data_stored":160885482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":161738447},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550068,"kb_used_data":301860,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93821772,"statfs":{"total":96636764160,"available":96073494528,"internally_reserved":0,"allocated":309104640,"data_stored":308947690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309379072,"data_stored":309379072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161689600,"data_stored":161689600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":307478528,"data_stored":307478528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310329344,"data_stored":310329344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:05:57.651 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 154 20 76912 76892 3.90021 2.20312 0.0705085 0.0400494 2024-09-15T21:05:58.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 155 20 77216 77196 3.89037 2.375 0.0600465 0.0401505 2024-09-15T21:05:59.648 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 156 20 77527 77507 3.88101 2.42969 0.0877231 0.0402445 2024-09-15T21:06:00.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:00.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 157 20 77826 77806 3.87117 2.33594 0.0658768 0.0403476 2024-09-15T21:06:00.937 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:00.937 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:00.965 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-15T21:05:59.964911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630785704,"num_objects":77005,"num_object_clones":0,"num_object_copies":308020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77032,"num_write_kb":616138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77059,"ondisk_log_size":77059,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1988308,"kb_used_data":1086096,"kb_used_omap":14,"kb_used_meta":902129,"kb_avail":375499052,"statfs":{"total":386547056640,"available":384511029248,"internally_reserved":0,"allocated":1112162304,"data_stored":1111534504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":923780658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,8,9,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29532160,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":28840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007051"},"pg_stats":[{"pgid":"1.f","version":"31'4798","reported_seq":4839,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.711915+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:58.711915+0000","last_peered":"2024-09-15T21:05:58.711915+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:58.711915+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39305216,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":38384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4846","reported_seq":4909,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.869104+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:57.869104+0000","last_peered":"2024-09-15T21:05:57.869104+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:57.869104+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39662860,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4688","reported_seq":4762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.286537+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:56.286537+0000","last_peered":"2024-09-15T21:05:56.286537+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:56.286537+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38378118,"num_objects":4685,"num_object_clones":0,"num_object_copies":18740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4687,"num_write_kb":37486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4815","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.965183+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:57.965183+0000","last_peered":"2024-09-15T21:05:57.965183+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:57.965183+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408908,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4915","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.647041+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:58.647041+0000","last_peered":"2024-09-15T21:05:58.647041+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:58.647041+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236300,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4806","reported_seq":4846,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.658209+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:58.658209+0000","last_peered":"2024-09-15T21:05:58.658209+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:58.658209+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4933","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.991669+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:57.991669+0000","last_peered":"2024-09-15T21:05:57.991669+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:57.991669+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359180,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4923","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.014476+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:58.014476+0000","last_peered":"2024-09-15T21:05:58.014476+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:58.014476+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285452,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4812","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.560682+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:58.560682+0000","last_peered":"2024-09-15T21:05:58.560682+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:58.560682+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4812,"log_dups_size":0,"ondisk_log_size":4812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39348760,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4778","reported_seq":4854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.386545+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:56.386545+0000","last_peered":"2024-09-15T21:05:56.386545+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:56.386545+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122188,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4776,"num_write_kb":38206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4787","reported_seq":7663,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.692752+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:58.692752+0000","last_peered":"2024-09-15T21:05:58.692752+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:58.692752+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39169938,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4738","reported_seq":4773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.612298+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:58.612298+0000","last_peered":"2024-09-15T21:05:58.612298+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:58.612298+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804102,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4778","reported_seq":4853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.359898+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:56.359898+0000","last_peered":"2024-09-15T21:05:56.359898+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:56.359898+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39131782,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4835","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.654588+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:58.654588+0000","last_peered":"2024-09-15T21:05:58.654588+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:58.654588+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39572748,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4783","reported_seq":4859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.125470+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:56.125470+0000","last_peered":"2024-09-15T21:05:56.125470+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:56.125470+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154956,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4824","reported_seq":4888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.953336+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:57.953336+0000","last_peered":"2024-09-15T21:05:57.953336+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:57.953336+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":630785704,"num_objects":77005,"num_object_clones":0,"num_object_copies":308020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77032,"num_write_kb":616138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113571328,"data_stored":1113571328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77059,"ondisk_log_size":77059,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":308772,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":316182528,"data_stored":316025578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549580,"kb_used_data":305976,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93822260,"statfs":{"total":96636764160,"available":96073994240,"internally_reserved":0,"allocated":313319424,"data_stored":313162474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325816,"kb_used_data":163688,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94046024,"statfs":{"total":96636764160,"available":96303128576,"internally_reserved":0,"allocated":167616512,"data_stored":167459562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560028,"kb_used_data":307660,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93811812,"statfs":{"total":96636764160,"available":96063295488,"internally_reserved":0,"allocated":315043840,"data_stored":314886890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315469824,"data_stored":315469824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168067072,"data_stored":168067072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313655296,"data_stored":313655296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316379136,"data_stored":316379136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:00.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:01.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:01.574 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:01.597 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-15T21:05:59.964911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":630785704,"num_objects":77005,"num_object_clones":0,"num_object_copies":308020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77032,"num_write_kb":616138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77059,"ondisk_log_size":77059,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1988308,"kb_used_data":1086096,"kb_used_omap":14,"kb_used_meta":902129,"kb_avail":375499052,"statfs":{"total":386547056640,"available":384511029248,"internally_reserved":0,"allocated":1112162304,"data_stored":1111534504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":923780658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,8,9,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29532160,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3605,"num_write_kb":28840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":1531904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007051"},"pg_stats":[{"pgid":"1.f","version":"31'4798","reported_seq":4839,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.711915+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:58.711915+0000","last_peered":"2024-09-15T21:05:58.711915+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:58.711915+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39305216,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":38384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4846","reported_seq":4909,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.869104+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:57.869104+0000","last_peered":"2024-09-15T21:05:57.869104+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:57.869104+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39662860,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4688","reported_seq":4762,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.286537+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:05:56.286537+0000","last_peered":"2024-09-15T21:05:56.286537+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:05:56.286537+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38378118,"num_objects":4685,"num_object_clones":0,"num_object_copies":18740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4687,"num_write_kb":37486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4815","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.965183+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:57.965183+0000","last_peered":"2024-09-15T21:05:57.965183+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:57.965183+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408908,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4915","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.647041+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:58.647041+0000","last_peered":"2024-09-15T21:05:58.647041+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:58.647041+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236300,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4806","reported_seq":4846,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.658209+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:58.658209+0000","last_peered":"2024-09-15T21:05:58.658209+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:58.658209+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4933","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.991669+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:57.991669+0000","last_peered":"2024-09-15T21:05:57.991669+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:57.991669+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359180,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4923","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.014476+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:58.014476+0000","last_peered":"2024-09-15T21:05:58.014476+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:58.014476+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285452,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4812","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.560682+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:58.560682+0000","last_peered":"2024-09-15T21:05:58.560682+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:58.560682+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4812,"log_dups_size":0,"ondisk_log_size":4812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39348760,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4778","reported_seq":4854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.386545+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:05:56.386545+0000","last_peered":"2024-09-15T21:05:56.386545+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:05:56.386545+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39122188,"num_objects":4776,"num_object_clones":0,"num_object_copies":19104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4776,"num_write_kb":38206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2441}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4787","reported_seq":7663,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.692752+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:58.692752+0000","last_peered":"2024-09-15T21:05:58.692752+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:58.692752+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39169938,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4738","reported_seq":4773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.612298+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:58.612298+0000","last_peered":"2024-09-15T21:05:58.612298+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:58.612298+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804102,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4778","reported_seq":4853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.359898+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:05:56.359898+0000","last_peered":"2024-09-15T21:05:56.359898+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:05:56.359898+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4778,"log_dups_size":0,"ondisk_log_size":4778,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39131782,"num_objects":4777,"num_object_clones":0,"num_object_copies":19108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4777,"num_write_kb":38215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4835","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.654588+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:58.654588+0000","last_peered":"2024-09-15T21:05:58.654588+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:58.654588+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39572748,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4783","reported_seq":4859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:56.125470+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:05:56.125470+0000","last_peered":"2024-09-15T21:05:56.125470+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:05:56.125470+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39154956,"num_objects":4780,"num_object_clones":0,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4781,"num_write_kb":38245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2499}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4824","reported_seq":4888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.953336+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:57.953336+0000","last_peered":"2024-09-15T21:05:57.953336+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:57.953336+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":630785704,"num_objects":77005,"num_object_clones":0,"num_object_copies":308020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77032,"num_write_kb":616138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1113571328,"data_stored":1113571328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77059,"ondisk_log_size":77059,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":308772,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":316182528,"data_stored":316025578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549580,"kb_used_data":305976,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":93822260,"statfs":{"total":96636764160,"available":96073994240,"internally_reserved":0,"allocated":313319424,"data_stored":313162474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051609,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325816,"kb_used_data":163688,"kb_used_omap":4,"kb_used_meta":162107,"kb_avail":94046024,"statfs":{"total":96636764160,"available":96303128576,"internally_reserved":0,"allocated":167616512,"data_stored":167459562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":165998287},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560028,"kb_used_data":307660,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93811812,"statfs":{"total":96636764160,"available":96063295488,"internally_reserved":0,"allocated":315043840,"data_stored":314886890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315469824,"data_stored":315469824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168067072,"data_stored":168067072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313655296,"data_stored":313655296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316379136,"data_stored":316379136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:01.598 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4798", 'reported_seq': 4839, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.711915+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:05:58.711915+0000', 'last_peered': '2024-09-15T21:05:58.711915+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:05:58.711915+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4798, 'log_dups_size': 0, 'ondisk_log_size': 4798, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39305216, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4798, 'num_write_kb': 38384, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4846", 'reported_seq': 4909, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:57.869104+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:05:57.869104+0000', 'last_peered': '2024-09-15T21:05:57.869104+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:05:57.869104+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4846, 'log_dups_size': 0, 'ondisk_log_size': 4846, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39662860, 'num_objects': 4842, 'num_object_clones': 0, 'num_object_copies': 19368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4842, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4844, 'num_write_kb': 38748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4688", 'reported_seq': 4762, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:56.286537+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:05:56.286537+0000', 'last_peered': '2024-09-15T21:05:56.286537+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:05:56.286537+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4688, 'log_dups_size': 0, 'ondisk_log_size': 4688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38378118, 'num_objects': 4685, 'num_object_clones': 0, 'num_object_copies': 18740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4687, 'num_write_kb': 37486, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4815", 'reported_seq': 4879, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:57.965183+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:05:57.965183+0000', 'last_peered': '2024-09-15T21:05:57.965183+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:05:57.965183+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4815, 'log_dups_size': 0, 'ondisk_log_size': 4815, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39408908, 'num_objects': 4811, 'num_object_clones': 0, 'num_object_copies': 19244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4811, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4813, 'num_write_kb': 38493, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2477}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:01.599 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'4915", 'reported_seq': 4966, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.647041+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:05:58.647041+0000', 'last_peered': '2024-09-15T21:05:58.647041+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:05:58.647041+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4915, 'log_dups_size': 0, 'ondisk_log_size': 4915, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40236300, 'num_objects': 4912, 'num_object_clones': 0, 'num_object_copies': 19648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4913, 'num_write_kb': 39301, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2601}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4806", 'reported_seq': 4846, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.658209+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:05:58.658209+0000', 'last_peered': '2024-09-15T21:05:58.658209+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:05:58.658209+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4806, 'log_dups_size': 0, 'ondisk_log_size': 4806, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39370752, 'num_objects': 4806, 'num_object_clones': 0, 'num_object_copies': 19224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4806, 'num_write_kb': 38448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2490}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'4933", 'reported_seq': 4998, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:57.991669+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:05:57.991669+0000', 'last_peered': '2024-09-15T21:05:57.991669+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:05:57.991669+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4933, 'log_dups_size': 0, 'ondisk_log_size': 4933, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40359180, 'num_objects': 4927, 'num_object_clones': 0, 'num_object_copies': 19708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4931, 'num_write_kb': 39428, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2561}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'4923", 'reported_seq': 4987, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.014476+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:05:58.014476+0000', 'last_peered': '2024-09-15T21:05:58.014476+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:05:58.014476+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4923, 'log_dups_size': 0, 'ondisk_log_size': 4923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40285452, 'num_objects': 4918, 'num_object_clones': 0, 'num_object_copies': 19672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4921, 'num_write_kb': 39356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2444}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4812", 'reported_seq': 4867, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.560682+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:05:58.560682+0000', 'last_peered': '2024-09-15T21:05:58.560682+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:05:58.560682+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4812, 'log_dups_size': 0, 'ondisk_log_size': 4812, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39348760, 'num_objects': 4804, 'num_object_clones': 0, 'num_object_copies': 19216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4809, 'num_write_kb': 38449, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2472}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:01.600 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4778", 'reported_seq': 4854, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:56.386545+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:05:56.386545+0000', 'last_peered': '2024-09-15T21:05:56.386545+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:05:56.386545+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4778, 'log_dups_size': 0, 'ondisk_log_size': 4778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39122188, 'num_objects': 4776, 'num_object_clones': 0, 'num_object_copies': 19104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4776, 'num_write_kb': 38206, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2441}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4787", 'reported_seq': 7663, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.692752+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:05:58.692752+0000', 'last_peered': '2024-09-15T21:05:58.692752+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:05:58.692752+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4787, 'log_dups_size': 0, 'ondisk_log_size': 4787, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39169938, 'num_objects': 4782, 'num_object_clones': 0, 'num_object_copies': 19128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 853, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4784, 'num_write_kb': 38260, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1413, 'num_bytes_recovered': 11526144, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3929}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4738", 'reported_seq': 4773, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.612298+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:05:58.612298+0000', 'last_peered': '2024-09-15T21:05:58.612298+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:05:58.612298+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4738, 'log_dups_size': 0, 'ondisk_log_size': 4738, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38804102, 'num_objects': 4737, 'num_object_clones': 0, 'num_object_copies': 18948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4737, 'num_write_kb': 37895, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2481}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4778", 'reported_seq': 4853, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:56.359898+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:05:56.359898+0000', 'last_peered': '2024-09-15T21:05:56.359898+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:05:56.359898+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4778, 'log_dups_size': 0, 'ondisk_log_size': 4778, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39131782, 'num_objects': 4777, 'num_object_clones': 0, 'num_object_copies': 19108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4777, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4777, 'num_write_kb': 38215, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4835", 'reported_seq': 4886, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:58.654588+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:05:58.654588+0000', 'last_peered': '2024-09-15T21:05:58.654588+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:05:58.654588+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4835, 'log_dups_size': 0, 'ondisk_log_size': 4835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39572748, 'num_objects': 4831, 'num_object_clones': 0, 'num_object_copies': 19324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4831, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4833, 'num_write_kb': 38660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2552}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:01.601 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4783", 'reported_seq': 4859, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:56.125470+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:05:56.125470+0000', 'last_peered': '2024-09-15T21:05:56.125470+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:05:56.125470+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4783, 'log_dups_size': 0, 'ondisk_log_size': 4783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39154956, 'num_objects': 4780, 'num_object_clones': 0, 'num_object_copies': 19120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4781, 'num_write_kb': 38245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2499}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:01.602 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:01.602 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4824", 'reported_seq': 4888, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:05:57.953336+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:05:57.953336+0000', 'last_peered': '2024-09-15T21:05:57.953336+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:05:57.953336+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4824, 'log_dups_size': 0, 'ondisk_log_size': 4824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39474444, 'num_objects': 4819, 'num_object_clones': 0, 'num_object_copies': 19276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4822, 'num_write_kb': 38564, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:01.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:01.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 158 20 78098 78078 3.86011 2.125 0.068906 0.0404629 2024-09-15T21:06:02.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:02.325 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35676/308020 objects degraded (11.582%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":228,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77005,"data_bytes":630785704,"bytes_used":2036027392,"bytes_avail":384511029248,"bytes_total":386547056640,"degraded_objects":35676,"degraded_total":308020,"degraded_ratio":0.11582364781507694,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":127583,"recovering_keys_per_sec":0,"num_objects_recovered":190,"num_bytes_recovered":1531904,"num_keys_recovered":0,"write_bytes_sec":2459568,"read_op_per_sec":0,"write_op_per_sec":300},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:02.356 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:02.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:02.651 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 159 20 78366 78346 3.849 2.09375 0.0437642 0.0405797 2024-09-15T21:06:03.032 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:03.032 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:03.056 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":136,"stamp":"2024-09-15T21:06:01.965827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633988776,"num_objects":77396,"num_object_clones":0,"num_object_copies":309584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77423,"num_write_kb":619266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77450,"ondisk_log_size":77450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2008744,"kb_used_data":1098216,"kb_used_omap":14,"kb_used_meta":910449,"kb_avail":375478616,"statfs":{"total":386547056640,"available":384490102784,"internally_reserved":0,"allocated":1124573184,"data_stored":1123945384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":932300338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,12,11,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3326,"num_write_kb":26608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1024000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007321"},"pg_stats":[{"pgid":"1.f","version":"31'4798","reported_seq":4839,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.711915+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:05:58.711915+0000","last_peered":"2024-09-15T21:05:58.711915+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:05:58.711915+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39305216,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4798,"num_write_kb":38384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4846","reported_seq":4909,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.869104+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:05:57.869104+0000","last_peered":"2024-09-15T21:05:57.869104+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:05:57.869104+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39662860,"num_objects":4842,"num_object_clones":0,"num_object_copies":19368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4844,"num_write_kb":38748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4801","reported_seq":4874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.458099+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:01.458099+0000","last_peered":"2024-09-15T21:06:01.458099+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:01.458099+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303814,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4800,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4815","reported_seq":4879,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.965183+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:05:57.965183+0000","last_peered":"2024-09-15T21:05:57.965183+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:05:57.965183+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4815,"log_dups_size":0,"ondisk_log_size":4815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39408908,"num_objects":4811,"num_object_clones":0,"num_object_copies":19244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4813,"num_write_kb":38493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2477}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'4915","reported_seq":4966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.647041+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:05:58.647041+0000","last_peered":"2024-09-15T21:05:58.647041+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:05:58.647041+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40236300,"num_objects":4912,"num_object_clones":0,"num_object_copies":19648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4913,"num_write_kb":39301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4806","reported_seq":4846,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.658209+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:05:58.658209+0000","last_peered":"2024-09-15T21:05:58.658209+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:05:58.658209+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39370752,"num_objects":4806,"num_object_clones":0,"num_object_copies":19224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":38448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2490}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'4933","reported_seq":4998,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.991669+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:05:57.991669+0000","last_peered":"2024-09-15T21:05:57.991669+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:05:57.991669+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40359180,"num_objects":4927,"num_object_clones":0,"num_object_copies":19708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":39428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'4923","reported_seq":4987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.014476+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:05:58.014476+0000","last_peered":"2024-09-15T21:05:58.014476+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:05:58.014476+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40285452,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4921,"num_write_kb":39356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2444}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4812","reported_seq":4867,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.560682+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:05:58.560682+0000","last_peered":"2024-09-15T21:05:58.560682+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:05:58.560682+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4812,"log_dups_size":0,"ondisk_log_size":4812,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39348760,"num_objects":4804,"num_object_clones":0,"num_object_copies":19216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4809,"num_write_kb":38449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.447846+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:01.447846+0000","last_peered":"2024-09-15T21:06:01.447846+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:01.447846+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39900428,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4787","reported_seq":7663,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.692752+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:05:58.692752+0000","last_peered":"2024-09-15T21:05:58.692752+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:05:58.692752+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39169938,"num_objects":4782,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":853,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4784,"num_write_kb":38260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1413,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3929}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4738","reported_seq":4773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.612298+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:05:58.612298+0000","last_peered":"2024-09-15T21:05:58.612298+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:05:58.612298+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38804102,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":37895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2481}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.363022+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:01.363022+0000","last_peered":"2024-09-15T21:06:01.363022+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:01.363022+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39910022,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4872,"num_write_kb":38975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4835","reported_seq":4886,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:58.654588+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:05:58.654588+0000","last_peered":"2024-09-15T21:05:58.654588+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:05:58.654588+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39572748,"num_objects":4831,"num_object_clones":0,"num_object_copies":19324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4833,"num_write_kb":38660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2552}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4871","reported_seq":4949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.376280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:01.376280+0000","last_peered":"2024-09-15T21:06:01.376280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:01.376280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875852,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2587}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4824","reported_seq":4888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:05:57.953336+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:05:57.953336+0000","last_peered":"2024-09-15T21:05:57.953336+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:05:57.953336+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39474444,"num_objects":4819,"num_object_clones":0,"num_object_copies":19276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4822,"num_write_kb":38564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":633988776,"num_objects":77396,"num_object_clones":0,"num_object_copies":309584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":77423,"num_write_kb":619266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1437,"num_bytes_recovered":11526144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1125933056,"data_stored":1125933056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77450,"ondisk_log_size":77450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552884,"kb_used_data":308772,"kb_used_omap":4,"kb_used_meta":244091,"kb_avail":93818956,"statfs":{"total":96636764160,"available":96070610944,"internally_reserved":0,"allocated":316182528,"data_stored":316025578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":249949904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559672,"kb_used_data":311912,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93812168,"statfs":{"total":96636764160,"available":96063660032,"internally_reserved":0,"allocated":319397888,"data_stored":319240938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336160,"kb_used_data":169872,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94035680,"statfs":{"total":96636764160,"available":96292536320,"internally_reserved":0,"allocated":173948928,"data_stored":173791978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560028,"kb_used_data":307660,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93811812,"statfs":{"total":96636764160,"available":96063295488,"internally_reserved":0,"allocated":315043840,"data_stored":314886890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":315469824,"data_stored":315469824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174411776,"data_stored":174411776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319672320,"data_stored":319672320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316379136,"data_stored":316379136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:03.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:06:03.647022+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0406836 2024-09-15T21:06:03.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:06:03.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 160 20 78666 78646 3.8396 2.34375 0.0436194 0.0406836 2024-09-15T21:06:04.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 161 20 78966 78946 3.8303 2.34375 0.101614 0.0407805 2024-09-15T21:06:05.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 162 20 79274 79254 3.82151 2.40625 0.0762852 0.040873 2024-09-15T21:06:06.058 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:06.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 163 20 79557 79537 3.81163 2.21094 0.0704215 0.0409804 2024-09-15T21:06:06.681 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:06.683 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:06.704 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-15T21:06:05.967443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642778792,"num_objects":78469,"num_object_clones":0,"num_object_copies":313876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78496,"num_write_kb":627850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78523,"ondisk_log_size":78523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2028788,"kb_used_data":1109940,"kb_used_omap":14,"kb_used_meta":918769,"kb_avail":375458572,"statfs":{"total":386547056640,"available":384469577728,"internally_reserved":0,"allocated":1136578560,"data_stored":1135950760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":940820018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,11,10,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24190976,"num_objects":2953,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":23624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007344"},"pg_stats":[{"pgid":"1.f","version":"31'4879","reported_seq":4920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.721161+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:03.721161+0000","last_peered":"2024-09-15T21:06:03.721161+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:03.721161+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4929","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.842839+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:02.842839+0000","last_peered":"2024-09-15T21:06:02.842839+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:02.842839+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40342796,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4927,"num_write_kb":39412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4801","reported_seq":4874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.458099+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:01.458099+0000","last_peered":"2024-09-15T21:06:01.458099+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:01.458099+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303814,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4800,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4910","reported_seq":4974,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.932265+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:02.932265+0000","last_peered":"2024-09-15T21:06:02.932265+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:02.932265+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187148,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5007","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.761076+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:03.761076+0000","last_peered":"2024-09-15T21:06:03.761076+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:03.761076+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989964,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4896","reported_seq":4936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.607116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:03.607116+0000","last_peered":"2024-09-15T21:06:03.607116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:03.607116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5030","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.939227+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:02.939227+0000","last_peered":"2024-09-15T21:06:02.939227+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:02.939227+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41153804,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":40204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2658}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5023","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.993752+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:02.993752+0000","last_peered":"2024-09-15T21:06:02.993752+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:02.993752+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41104652,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4898","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.746255+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:03.746255+0000","last_peered":"2024-09-15T21:06:03.746255+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:03.746255+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40053272,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4895,"num_write_kb":39137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.447846+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:01.447846+0000","last_peered":"2024-09-15T21:06:01.447846+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:01.447846+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39900428,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4873","reported_seq":7879,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.779421+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:03.779421+0000","last_peered":"2024-09-15T21:06:03.779421+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:03.779421+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39874450,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4870,"num_write_kb":38948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4835","reported_seq":4870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.719322+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:03.719322+0000","last_peered":"2024-09-15T21:06:03.719322+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:03.719322+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39598726,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4834,"num_write_kb":38671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2578}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.363022+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:01.363022+0000","last_peered":"2024-09-15T21:06:01.363022+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:01.363022+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39910022,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4872,"num_write_kb":38975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4907","reported_seq":4959,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.503696+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:03.503696+0000","last_peered":"2024-09-15T21:06:03.503696+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:03.503696+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40162572,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4905,"num_write_kb":39236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4871","reported_seq":4949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.376280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:01.376280+0000","last_peered":"2024-09-15T21:06:01.376280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:01.376280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875852,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2587}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4918","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.854620+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:02.854620+0000","last_peered":"2024-09-15T21:06:02.854620+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:02.854620+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244492,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642778792,"num_objects":78469,"num_object_clones":0,"num_object_copies":313876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78496,"num_write_kb":627850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1137860608,"data_stored":1137860608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78523,"ondisk_log_size":78523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562968,"kb_used_data":314696,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93808872,"statfs":{"total":96636764160,"available":96060284928,"internally_reserved":0,"allocated":322248704,"data_stored":322091754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559672,"kb_used_data":311912,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93812168,"statfs":{"total":96636764160,"available":96063660032,"internally_reserved":0,"allocated":319397888,"data_stored":319240938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336160,"kb_used_data":169872,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94035680,"statfs":{"total":96636764160,"available":96292536320,"internally_reserved":0,"allocated":173948928,"data_stored":173791978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569988,"kb_used_data":313460,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93801852,"statfs":{"total":96636764160,"available":96053096448,"internally_reserved":0,"allocated":320983040,"data_stored":320826090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321417216,"data_stored":321417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174411776,"data_stored":174411776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319672320,"data_stored":319672320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322359296,"data_stored":322359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:06.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:07.279 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:07.280 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:07.306 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-09-15T21:06:05.967443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":642778792,"num_objects":78469,"num_object_clones":0,"num_object_copies":313876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78496,"num_write_kb":627850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78523,"ondisk_log_size":78523,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2028788,"kb_used_data":1109940,"kb_used_omap":14,"kb_used_meta":918769,"kb_avail":375458572,"statfs":{"total":386547056640,"available":384469577728,"internally_reserved":0,"allocated":1136578560,"data_stored":1135950760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":940820018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,11,10,6,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24190976,"num_objects":2953,"num_object_clones":0,"num_object_copies":11812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2953,"num_write_kb":23624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007344"},"pg_stats":[{"pgid":"1.f","version":"31'4879","reported_seq":4920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.721161+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:03.721161+0000","last_peered":"2024-09-15T21:06:03.721161+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:03.721161+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4929","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.842839+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:02.842839+0000","last_peered":"2024-09-15T21:06:02.842839+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:02.842839+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40342796,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4927,"num_write_kb":39412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4801","reported_seq":4874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.458099+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:01.458099+0000","last_peered":"2024-09-15T21:06:01.458099+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:01.458099+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4801,"log_dups_size":0,"ondisk_log_size":4801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39303814,"num_objects":4798,"num_object_clones":0,"num_object_copies":19192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4800,"num_write_kb":38390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2540}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4910","reported_seq":4974,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.932265+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:02.932265+0000","last_peered":"2024-09-15T21:06:02.932265+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:02.932265+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187148,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5007","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.761076+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:03.761076+0000","last_peered":"2024-09-15T21:06:03.761076+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:03.761076+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989964,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4896","reported_seq":4936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.607116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:03.607116+0000","last_peered":"2024-09-15T21:06:03.607116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:03.607116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5030","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.939227+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:02.939227+0000","last_peered":"2024-09-15T21:06:02.939227+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:02.939227+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41153804,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":40204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2658}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5023","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.993752+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:02.993752+0000","last_peered":"2024-09-15T21:06:02.993752+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:02.993752+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41104652,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4898","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.746255+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:03.746255+0000","last_peered":"2024-09-15T21:06:03.746255+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:03.746255+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40053272,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4895,"num_write_kb":39137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.447846+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:01.447846+0000","last_peered":"2024-09-15T21:06:01.447846+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:01.447846+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39900428,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4871,"num_write_kb":38966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2536}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4873","reported_seq":7879,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.779421+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:03.779421+0000","last_peered":"2024-09-15T21:06:03.779421+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:03.779421+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39874450,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4870,"num_write_kb":38948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4835","reported_seq":4870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.719322+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:03.719322+0000","last_peered":"2024-09-15T21:06:03.719322+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:03.719322+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39598726,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4834,"num_write_kb":38671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2578}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4873","reported_seq":4948,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.363022+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:01.363022+0000","last_peered":"2024-09-15T21:06:01.363022+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:01.363022+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39910022,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4872,"num_write_kb":38975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4907","reported_seq":4959,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.503696+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:03.503696+0000","last_peered":"2024-09-15T21:06:03.503696+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:03.503696+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40162572,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4905,"num_write_kb":39236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4871","reported_seq":4949,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:01.376280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:01.376280+0000","last_peered":"2024-09-15T21:06:01.376280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:01.376280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4871,"log_dups_size":0,"ondisk_log_size":4871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39875852,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4869,"num_write_kb":38949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2587}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4918","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.854620+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:02.854620+0000","last_peered":"2024-09-15T21:06:02.854620+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:02.854620+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244492,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":642778792,"num_objects":78469,"num_object_clones":0,"num_object_copies":313876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78496,"num_write_kb":627850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1137860608,"data_stored":1137860608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78523,"ondisk_log_size":78523,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562968,"kb_used_data":314696,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93808872,"statfs":{"total":96636764160,"available":96060284928,"internally_reserved":0,"allocated":322248704,"data_stored":322091754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":559672,"kb_used_data":311912,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":93812168,"statfs":{"total":96636764160,"available":96063660032,"internally_reserved":0,"allocated":319397888,"data_stored":319240938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051610,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":336160,"kb_used_data":169872,"kb_used_omap":4,"kb_used_meta":166267,"kb_avail":94035680,"statfs":{"total":96636764160,"available":96292536320,"internally_reserved":0,"allocated":173948928,"data_stored":173791978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":170258127},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,5,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569988,"kb_used_data":313460,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93801852,"statfs":{"total":96636764160,"available":96053096448,"internally_reserved":0,"allocated":320983040,"data_stored":320826090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321417216,"data_stored":321417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":174411776,"data_stored":174411776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319672320,"data_stored":319672320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322359296,"data_stored":322359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4879", 'reported_seq': 4920, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.721161+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:03.721161+0000', 'last_peered': '2024-09-15T21:06:03.721161+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:03.721161+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4879, 'log_dups_size': 0, 'ondisk_log_size': 4879, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39968768, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4879, 'num_write_kb': 39032, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'4929", 'reported_seq': 4992, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:02.842839+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:02.842839+0000', 'last_peered': '2024-09-15T21:06:02.842839+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:02.842839+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4929, 'log_dups_size': 0, 'ondisk_log_size': 4929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40342796, 'num_objects': 4925, 'num_object_clones': 0, 'num_object_copies': 19700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4925, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4927, 'num_write_kb': 39412, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:07.307 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4801", 'reported_seq': 4874, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:01.458099+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:01.458099+0000', 'last_peered': '2024-09-15T21:06:01.458099+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:01.458099+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4801, 'log_dups_size': 0, 'ondisk_log_size': 4801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39303814, 'num_objects': 4798, 'num_object_clones': 0, 'num_object_copies': 19192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4798, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4800, 'num_write_kb': 38390, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2540}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'4910", 'reported_seq': 4974, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:02.932265+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:02.932265+0000', 'last_peered': '2024-09-15T21:06:02.932265+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:02.932265+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4910, 'log_dups_size': 0, 'ondisk_log_size': 4910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40187148, 'num_objects': 4906, 'num_object_clones': 0, 'num_object_copies': 19624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4906, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4908, 'num_write_kb': 39253, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2572}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5007", 'reported_seq': 5058, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.761076+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:03.761076+0000', 'last_peered': '2024-09-15T21:06:03.761076+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:03.761076+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5007, 'log_dups_size': 0, 'ondisk_log_size': 5007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40989964, 'num_objects': 5004, 'num_object_clones': 0, 'num_object_copies': 20016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 40037, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2693}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4896", 'reported_seq': 4936, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.607116+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:03.607116+0000', 'last_peered': '2024-09-15T21:06:03.607116+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:03.607116+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4896, 'log_dups_size': 0, 'ondisk_log_size': 4896, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40108032, 'num_objects': 4896, 'num_object_clones': 0, 'num_object_copies': 19584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4896, 'num_write_kb': 39168, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2580}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5030", 'reported_seq': 5095, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:02.939227+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:02.939227+0000', 'last_peered': '2024-09-15T21:06:02.939227+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:02.939227+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5030, 'log_dups_size': 0, 'ondisk_log_size': 5030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41153804, 'num_objects': 5024, 'num_object_clones': 0, 'num_object_copies': 20096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5028, 'num_write_kb': 40204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2658}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5023", 'reported_seq': 5087, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:02.993752+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:02.993752+0000', 'last_peered': '2024-09-15T21:06:02.993752+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:02.993752+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5023, 'log_dups_size': 0, 'ondisk_log_size': 5023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41104652, 'num_objects': 5018, 'num_object_clones': 0, 'num_object_copies': 20072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5021, 'num_write_kb': 40156, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:07.308 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4898", 'reported_seq': 4952, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.746255+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:03.746255+0000', 'last_peered': '2024-09-15T21:06:03.746255+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:03.746255+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4898, 'log_dups_size': 0, 'ondisk_log_size': 4898, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40053272, 'num_objects': 4890, 'num_object_clones': 0, 'num_object_copies': 19560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4890, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4895, 'num_write_kb': 39137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2558}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'4873", 'reported_seq': 4948, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:01.447846+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:01.447846+0000', 'last_peered': '2024-09-15T21:06:01.447846+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:01.447846+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4873, 'log_dups_size': 0, 'ondisk_log_size': 4873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39900428, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4871, 'num_write_kb': 38966, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2536}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4873", 'reported_seq': 7879, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.779421+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:03.779421+0000', 'last_peered': '2024-09-15T21:06:03.779421+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:03.779421+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4873, 'log_dups_size': 0, 'ondisk_log_size': 4873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39874450, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 788, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4870, 'num_write_kb': 38948, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1478, 'num_bytes_recovered': 12058624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4080}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4835", 'reported_seq': 4870, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.719322+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:03.719322+0000', 'last_peered': '2024-09-15T21:06:03.719322+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:03.719322+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4835, 'log_dups_size': 0, 'ondisk_log_size': 4835, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39598726, 'num_objects': 4834, 'num_object_clones': 0, 'num_object_copies': 19336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4834, 'num_write_kb': 38671, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2578}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'4873", 'reported_seq': 4948, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:01.363022+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:01.363022+0000', 'last_peered': '2024-09-15T21:06:01.363022+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:01.363022+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4873, 'log_dups_size': 0, 'ondisk_log_size': 4873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39910022, 'num_objects': 4872, 'num_object_clones': 0, 'num_object_copies': 19488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4872, 'num_write_kb': 38975, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:07.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'4907", 'reported_seq': 4959, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:03.503696+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:03.503696+0000', 'last_peered': '2024-09-15T21:06:03.503696+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:03.503696+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4907, 'log_dups_size': 0, 'ondisk_log_size': 4907, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40162572, 'num_objects': 4903, 'num_object_clones': 0, 'num_object_copies': 19612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4905, 'num_write_kb': 39236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:07.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:07.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'4871", 'reported_seq': 4949, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:01.376280+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:01.376280+0000', 'last_peered': '2024-09-15T21:06:01.376280+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:01.376280+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4871, 'log_dups_size': 0, 'ondisk_log_size': 4871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39875852, 'num_objects': 4868, 'num_object_clones': 0, 'num_object_copies': 19472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4868, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4869, 'num_write_kb': 38949, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2587}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:07.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:07.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'4918", 'reported_seq': 4982, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:02.854620+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:02.854620+0000', 'last_peered': '2024-09-15T21:06:02.854620+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:02.854620+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4918, 'log_dups_size': 0, 'ondisk_log_size': 4918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40244492, 'num_objects': 4913, 'num_object_clones': 0, 'num_object_copies': 19652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4916, 'num_write_kb': 39316, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:07.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:07.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 164 20 79821 79801 3.80096 2.0625 0.0825177 0.0410935 2024-09-15T21:06:08.035 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:08.035 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35611/313876 objects degraded (11.346%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":233,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78469,"data_bytes":642778792,"bytes_used":2077478912,"bytes_avail":384469577728,"bytes_total":386547056640,"degraded_objects":35611,"degraded_total":313876,"degraded_ratio":0.113455632160471,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":88692,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":2014681,"read_op_per_sec":0,"write_op_per_sec":245},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:08.069 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:08.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:08.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 165 20 80085 80065 3.79043 2.0625 0.0524388 0.0412099 2024-09-15T21:06:08.685 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:08.685 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:08.714 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-15T21:06:07.968339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645818024,"num_objects":78840,"num_object_clones":0,"num_object_copies":315360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78867,"num_write_kb":630818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78894,"ondisk_log_size":78894,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2066936,"kb_used_data":1121720,"kb_used_omap":14,"kb_used_meta":945137,"kb_avail":375420424,"statfs":{"total":386547056640,"available":384430514176,"internally_reserved":0,"allocated":1148641280,"data_stored":1148013480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":967820850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,6,10,7,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27230208,"num_objects":3324,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3324,"num_write_kb":26592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006813"},"pg_stats":[{"pgid":"1.f","version":"31'4879","reported_seq":4920,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.721161+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:03.721161+0000","last_peered":"2024-09-15T21:06:03.721161+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:03.721161+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39968768,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4879,"num_write_kb":39032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'4929","reported_seq":4992,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.842839+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:02.842839+0000","last_peered":"2024-09-15T21:06:02.842839+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:02.842839+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40342796,"num_objects":4925,"num_object_clones":0,"num_object_copies":19700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4927,"num_write_kb":39412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4895","reported_seq":4969,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:06.429000+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:06.429000+0000","last_peered":"2024-09-15T21:06:06.429000+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:06.429000+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40073862,"num_objects":4892,"num_object_clones":0,"num_object_copies":19568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4894,"num_write_kb":39142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'4910","reported_seq":4974,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.932265+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:02.932265+0000","last_peered":"2024-09-15T21:06:02.932265+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:02.932265+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4910,"log_dups_size":0,"ondisk_log_size":4910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40187148,"num_objects":4906,"num_object_clones":0,"num_object_copies":19624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4908,"num_write_kb":39253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2572}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5007","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.761076+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:03.761076+0000","last_peered":"2024-09-15T21:06:03.761076+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:03.761076+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40989964,"num_objects":5004,"num_object_clones":0,"num_object_copies":20016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2693}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4896","reported_seq":4936,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.607116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:03.607116+0000","last_peered":"2024-09-15T21:06:03.607116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:03.607116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4896,"log_dups_size":0,"ondisk_log_size":4896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":4896,"num_object_clones":0,"num_object_copies":19584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4896,"num_write_kb":39168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2580}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5030","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.939227+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:02.939227+0000","last_peered":"2024-09-15T21:06:02.939227+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:02.939227+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41153804,"num_objects":5024,"num_object_clones":0,"num_object_copies":20096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":40204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2658}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5023","reported_seq":5087,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.993752+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:02.993752+0000","last_peered":"2024-09-15T21:06:02.993752+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:02.993752+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41104652,"num_objects":5018,"num_object_clones":0,"num_object_copies":20072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5021,"num_write_kb":40156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4898","reported_seq":4952,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.746255+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:03.746255+0000","last_peered":"2024-09-15T21:06:03.746255+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:03.746255+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40053272,"num_objects":4890,"num_object_clones":0,"num_object_copies":19560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4895,"num_write_kb":39137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2558}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'4982","reported_seq":5058,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:06.330386+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:06.330386+0000","last_peered":"2024-09-15T21:06:06.330386+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:06.330386+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40793356,"num_objects":4980,"num_object_clones":0,"num_object_copies":19920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4980,"num_write_kb":39838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2645}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4873","reported_seq":7879,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.779421+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:03.779421+0000","last_peered":"2024-09-15T21:06:03.779421+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:03.779421+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39874450,"num_objects":4868,"num_object_clones":0,"num_object_copies":19472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4870,"num_write_kb":38948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1478,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4080}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4835","reported_seq":4870,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.719322+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:03.719322+0000","last_peered":"2024-09-15T21:06:03.719322+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:03.719322+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39598726,"num_objects":4834,"num_object_clones":0,"num_object_copies":19336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4834,"num_write_kb":38671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2578}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'4961","reported_seq":5036,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:06.367159+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:06.367159+0000","last_peered":"2024-09-15T21:06:06.367159+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:06.367159+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4961,"log_dups_size":0,"ondisk_log_size":4961,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40630918,"num_objects":4960,"num_object_clones":0,"num_object_copies":19840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4960,"num_write_kb":39679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'4907","reported_seq":4959,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:03.503696+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:03.503696+0000","last_peered":"2024-09-15T21:06:03.503696+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:03.503696+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40162572,"num_objects":4903,"num_object_clones":0,"num_object_copies":19612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4905,"num_write_kb":39236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'4951","reported_seq":5029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:06.246246+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:06.246246+0000","last_peered":"2024-09-15T21:06:06.246246+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:06.246246+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40531212,"num_objects":4948,"num_object_clones":0,"num_object_copies":19792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4949,"num_write_kb":39589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2667}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'4918","reported_seq":4982,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:02.854620+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:02.854620+0000","last_peered":"2024-09-15T21:06:02.854620+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:02.854620+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40244492,"num_objects":4913,"num_object_clones":0,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4916,"num_write_kb":39316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2598}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":645818024,"num_objects":78840,"num_object_clones":0,"num_object_copies":315360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78840,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78867,"num_write_kb":630818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1502,"num_bytes_recovered":12058624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1150169088,"data_stored":1150169088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78894,"ondisk_log_size":78894,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562968,"kb_used_data":314696,"kb_used_omap":4,"kb_used_meta":248251,"kb_avail":93808872,"statfs":{"total":96636764160,"available":96060284928,"internally_reserved":0,"allocated":322248704,"data_stored":322091754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":254209744},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569584,"kb_used_data":317664,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":93802256,"statfs":{"total":96636764160,"available":96053510144,"internally_reserved":0,"allocated":325287936,"data_stored":325130986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364396,"kb_used_data":175900,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":94007444,"statfs":{"total":96636764160,"available":96263622656,"internally_reserved":0,"allocated":180121600,"data_stored":179964650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192999119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569988,"kb_used_data":313460,"kb_used_omap":4,"kb_used_meta":256507,"kb_avail":93801852,"statfs":{"total":96636764160,"available":96053096448,"internally_reserved":0,"allocated":320983040,"data_stored":320826090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":262663886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321417216,"data_stored":321417216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180731904,"data_stored":180731904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":325660672,"data_stored":325660672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":322359296,"data_stored":322359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:09.649 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 166 20 80380 80360 3.78148 2.30469 0.119989 0.0413053 2024-09-15T21:06:10.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 167 20 80681 80661 3.77287 2.35156 0.0839143 0.0413992 2024-09-15T21:06:11.651 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 168 20 80990 80970 3.76478 2.41406 0.033995 0.0414869 2024-09-15T21:06:11.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:12.343 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:12.344 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:12.374 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-15T21:06:11.970062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657901224,"num_objects":80315,"num_object_clones":0,"num_object_copies":321260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80342,"num_write_kb":642618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079980,"kb_used_data":1138924,"kb_used_omap":14,"kb_used_meta":940977,"kb_avail":375407380,"statfs":{"total":386547056640,"available":384417157120,"internally_reserved":0,"allocated":1166258176,"data_stored":1165630376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":963561010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,6,3,12,10,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27115520,"num_objects":3310,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":26480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005151"},"pg_stats":[{"pgid":"1.f","version":"31'4976","reported_seq":5017,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.773645+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:08.773645+0000","last_peered":"2024-09-15T21:06:08.773645+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:08.773645+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40763392,"num_objects":4976,"num_object_clones":0,"num_object_copies":19904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5039","reported_seq":5102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.021549+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:08.021549+0000","last_peered":"2024-09-15T21:06:08.021549+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:08.021549+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243916,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5037,"num_write_kb":40292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4974","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.354546+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:11.354546+0000","last_peered":"2024-09-15T21:06:11.354546+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:11.354546+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40721030,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":39774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5004","reported_seq":5068,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:07.870014+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:07.870014+0000","last_peered":"2024-09-15T21:06:07.870014+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:07.870014+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40957196,"num_objects":5000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":40005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5100","reported_seq":5150,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.582200+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:08.582200+0000","last_peered":"2024-09-15T21:06:08.582200+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:08.582200+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41751820,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5098,"num_write_kb":40781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4988","reported_seq":5028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.837057+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:08.837057+0000","last_peered":"2024-09-15T21:06:08.837057+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:08.837057+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4988,"num_write_kb":39904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5120","reported_seq":5184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.034437+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:08.034437+0000","last_peered":"2024-09-15T21:06:08.034437+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:08.034437+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41891084,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5115","reported_seq":5179,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.018546+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:08.018546+0000","last_peered":"2024-09-15T21:06:08.018546+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:08.018546+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41858316,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4986","reported_seq":5041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.861490+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:08.861490+0000","last_peered":"2024-09-15T21:06:08.861490+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:08.861490+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40774168,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4983,"num_write_kb":39841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5079","reported_seq":5155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.424816+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:11.424816+0000","last_peered":"2024-09-15T21:06:11.424816+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:11.424816+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587980,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4948","reported_seq":8083,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.874706+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:08.874706+0000","last_peered":"2024-09-15T21:06:08.874706+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:08.874706+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40488850,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":39548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1543,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4919","reported_seq":4953,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.843218+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:08.843218+0000","last_peered":"2024-09-15T21:06:08.843218+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:08.843218+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40286854,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5043","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.176983+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:11.176983+0000","last_peered":"2024-09-15T21:06:11.176983+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:11.176983+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41302662,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":40335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2756}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5000","reported_seq":5050,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.753075+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:08.753075+0000","last_peered":"2024-09-15T21:06:08.753075+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:08.753075+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40924428,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4998,"num_write_kb":39980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5071","reported_seq":5149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.380370+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:11.380370+0000","last_peered":"2024-09-15T21:06:11.380370+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:11.380370+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514252,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":40549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2787}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5007","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:07.989377+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:07.989377+0000","last_peered":"2024-09-15T21:06:07.989377+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:07.989377+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40973580,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657901224,"num_objects":80315,"num_object_clones":0,"num_object_copies":321260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80342,"num_write_kb":642618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168003072,"data_stored":1168003072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572904,"kb_used_data":320472,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93798936,"statfs":{"total":96636764160,"available":96050110464,"internally_reserved":0,"allocated":328163328,"data_stored":328006378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579404,"kb_used_data":323324,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93792436,"statfs":{"total":96636764160,"available":96043454464,"internally_reserved":0,"allocated":331083776,"data_stored":330926826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364396,"kb_used_data":175900,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":94007444,"statfs":{"total":96636764160,"available":96263622656,"internally_reserved":0,"allocated":180121600,"data_stored":179964650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192999119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563276,"kb_used_data":319228,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93808564,"statfs":{"total":96636764160,"available":96059969536,"internally_reserved":0,"allocated":326889472,"data_stored":326732522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327438336,"data_stored":327438336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180731904,"data_stored":180731904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331522048,"data_stored":331522048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328310784,"data_stored":328310784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:12.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:12.651 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 169 20 81260 81240 3.75498 2.10938 0.0681269 0.0415979 2024-09-15T21:06:12.948 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:12.949 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:12.973 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-15T21:06:11.970062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657901224,"num_objects":80315,"num_object_clones":0,"num_object_copies":321260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80342,"num_write_kb":642618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2079980,"kb_used_data":1138924,"kb_used_omap":14,"kb_used_meta":940977,"kb_avail":375407380,"statfs":{"total":386547056640,"available":384417157120,"internally_reserved":0,"allocated":1166258176,"data_stored":1165630376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":963561010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,6,3,12,10,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27115520,"num_objects":3310,"num_object_clones":0,"num_object_copies":13240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3310,"num_write_kb":26480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005151"},"pg_stats":[{"pgid":"1.f","version":"31'4976","reported_seq":5017,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.773645+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:08.773645+0000","last_peered":"2024-09-15T21:06:08.773645+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:08.773645+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40763392,"num_objects":4976,"num_object_clones":0,"num_object_copies":19904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4976,"num_write_kb":39808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2700}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5039","reported_seq":5102,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.021549+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:08.021549+0000","last_peered":"2024-09-15T21:06:08.021549+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:08.021549+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243916,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5037,"num_write_kb":40292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2655}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4974","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.354546+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:11.354546+0000","last_peered":"2024-09-15T21:06:11.354546+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:11.354546+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40721030,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":39774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5004","reported_seq":5068,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:07.870014+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:07.870014+0000","last_peered":"2024-09-15T21:06:07.870014+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:07.870014+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40957196,"num_objects":5000,"num_object_clones":0,"num_object_copies":20000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":40005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2666}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5100","reported_seq":5150,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.582200+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:08.582200+0000","last_peered":"2024-09-15T21:06:08.582200+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:08.582200+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41751820,"num_objects":5097,"num_object_clones":0,"num_object_copies":20388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5098,"num_write_kb":40781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'4988","reported_seq":5028,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.837057+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:08.837057+0000","last_peered":"2024-09-15T21:06:08.837057+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:08.837057+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40861696,"num_objects":4988,"num_object_clones":0,"num_object_copies":19952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4988,"num_write_kb":39904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5120","reported_seq":5184,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.034437+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:08.034437+0000","last_peered":"2024-09-15T21:06:08.034437+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:08.034437+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5120,"log_dups_size":0,"ondisk_log_size":5120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41891084,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5118,"num_write_kb":40924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5115","reported_seq":5179,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.018546+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:08.018546+0000","last_peered":"2024-09-15T21:06:08.018546+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:08.018546+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41858316,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":40892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'4986","reported_seq":5041,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.861490+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:08.861490+0000","last_peered":"2024-09-15T21:06:08.861490+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:08.861490+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4986,"log_dups_size":0,"ondisk_log_size":4986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40774168,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4983,"num_write_kb":39841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2646}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5079","reported_seq":5155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.424816+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:11.424816+0000","last_peered":"2024-09-15T21:06:11.424816+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:11.424816+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587980,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'4948","reported_seq":8083,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.874706+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:08.874706+0000","last_peered":"2024-09-15T21:06:08.874706+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:08.874706+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40488850,"num_objects":4943,"num_object_clones":0,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":39548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1543,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4220}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'4919","reported_seq":4953,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.843218+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:08.843218+0000","last_peered":"2024-09-15T21:06:08.843218+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:08.843218+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40286854,"num_objects":4918,"num_object_clones":0,"num_object_copies":19672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4918,"num_write_kb":39343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2662}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5043","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.176983+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:11.176983+0000","last_peered":"2024-09-15T21:06:11.176983+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:11.176983+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41302662,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":40335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2756}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5000","reported_seq":5050,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:08.753075+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:08.753075+0000","last_peered":"2024-09-15T21:06:08.753075+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:08.753075+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5000,"log_dups_size":0,"ondisk_log_size":5000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40924428,"num_objects":4996,"num_object_clones":0,"num_object_copies":19984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4998,"num_write_kb":39980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2717}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5071","reported_seq":5149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.380370+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:11.380370+0000","last_peered":"2024-09-15T21:06:11.380370+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:11.380370+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514252,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":40549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2787}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5007","reported_seq":5071,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:07.989377+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:07.989377+0000","last_peered":"2024-09-15T21:06:07.989377+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:07.989377+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40973580,"num_objects":5002,"num_object_clones":0,"num_object_copies":20008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5005,"num_write_kb":40028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2687}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":657901224,"num_objects":80315,"num_object_clones":0,"num_object_copies":321260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80342,"num_write_kb":642618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12591104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1168003072,"data_stored":1168003072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572904,"kb_used_data":320472,"kb_used_omap":4,"kb_used_meta":252411,"kb_avail":93798936,"statfs":{"total":96636764160,"available":96050110464,"internally_reserved":0,"allocated":328163328,"data_stored":328006378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":258469584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,1,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579404,"kb_used_data":323324,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93792436,"statfs":{"total":96636764160,"available":96043454464,"internally_reserved":0,"allocated":331083776,"data_stored":330926826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051611,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":364396,"kb_used_data":175900,"kb_used_omap":4,"kb_used_meta":188475,"kb_avail":94007444,"statfs":{"total":96636764160,"available":96263622656,"internally_reserved":0,"allocated":180121600,"data_stored":179964650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":192999119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563276,"kb_used_data":319228,"kb_used_omap":4,"kb_used_meta":244027,"kb_avail":93808564,"statfs":{"total":96636764160,"available":96059969536,"internally_reserved":0,"allocated":326889472,"data_stored":326732522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":249884366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327438336,"data_stored":327438336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":180731904,"data_stored":180731904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331522048,"data_stored":331522048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328310784,"data_stored":328310784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'4976", 'reported_seq': 5017, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.773645+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:08.773645+0000', 'last_peered': '2024-09-15T21:06:08.773645+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:08.773645+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4976, 'log_dups_size': 0, 'ondisk_log_size': 4976, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40763392, 'num_objects': 4976, 'num_object_clones': 0, 'num_object_copies': 19904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4976, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4976, 'num_write_kb': 39808, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2700}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5039", 'reported_seq': 5102, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.021549+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:08.021549+0000', 'last_peered': '2024-09-15T21:06:08.021549+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:08.021549+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5039, 'log_dups_size': 0, 'ondisk_log_size': 5039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41243916, 'num_objects': 5035, 'num_object_clones': 0, 'num_object_copies': 20140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5037, 'num_write_kb': 40292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'4974", 'reported_seq': 5048, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:11.354546+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:11.354546+0000', 'last_peered': '2024-09-15T21:06:11.354546+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:11.354546+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4974, 'log_dups_size': 0, 'ondisk_log_size': 4974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40721030, 'num_objects': 4971, 'num_object_clones': 0, 'num_object_copies': 19884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4973, 'num_write_kb': 39774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2713}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5004", 'reported_seq': 5068, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:07.870014+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:07.870014+0000', 'last_peered': '2024-09-15T21:06:07.870014+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:07.870014+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5004, 'log_dups_size': 0, 'ondisk_log_size': 5004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40957196, 'num_objects': 5000, 'num_object_clones': 0, 'num_object_copies': 20000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5002, 'num_write_kb': 40005, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2666}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:12.974 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5100", 'reported_seq': 5150, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.582200+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:08.582200+0000', 'last_peered': '2024-09-15T21:06:08.582200+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:08.582200+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5100, 'log_dups_size': 0, 'ondisk_log_size': 5100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41751820, 'num_objects': 5097, 'num_object_clones': 0, 'num_object_copies': 20388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5097, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5098, 'num_write_kb': 40781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'4988", 'reported_seq': 5028, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.837057+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:08.837057+0000', 'last_peered': '2024-09-15T21:06:08.837057+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:08.837057+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4988, 'log_dups_size': 0, 'ondisk_log_size': 4988, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40861696, 'num_objects': 4988, 'num_object_clones': 0, 'num_object_copies': 19952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4988, 'num_write_kb': 39904, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5120", 'reported_seq': 5184, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.034437+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:08.034437+0000', 'last_peered': '2024-09-15T21:06:08.034437+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:08.034437+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5120, 'log_dups_size': 0, 'ondisk_log_size': 5120, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41891084, 'num_objects': 5114, 'num_object_clones': 0, 'num_object_copies': 20456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5118, 'num_write_kb': 40924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5115", 'reported_seq': 5179, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.018546+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:08.018546+0000', 'last_peered': '2024-09-15T21:06:08.018546+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:08.018546+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5115, 'log_dups_size': 0, 'ondisk_log_size': 5115, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41858316, 'num_objects': 5110, 'num_object_clones': 0, 'num_object_copies': 20440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5110, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5113, 'num_write_kb': 40892, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'4986", 'reported_seq': 5041, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.861490+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:08.861490+0000', 'last_peered': '2024-09-15T21:06:08.861490+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:08.861490+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4986, 'log_dups_size': 0, 'ondisk_log_size': 4986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40774168, 'num_objects': 4978, 'num_object_clones': 0, 'num_object_copies': 19912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4983, 'num_write_kb': 39841, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2646}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:12.975 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5079", 'reported_seq': 5155, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:11.424816+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:11.424816+0000', 'last_peered': '2024-09-15T21:06:11.424816+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:11.424816+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5079, 'log_dups_size': 0, 'ondisk_log_size': 5079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41587980, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5077, 'num_write_kb': 40614, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2742}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'4948", 'reported_seq': 8083, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.874706+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:08.874706+0000', 'last_peered': '2024-09-15T21:06:08.874706+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:08.874706+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4948, 'log_dups_size': 0, 'ondisk_log_size': 4948, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40488850, 'num_objects': 4943, 'num_object_clones': 0, 'num_object_copies': 19772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 723, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4945, 'num_write_kb': 39548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1543, 'num_bytes_recovered': 12591104, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4220}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'4919", 'reported_seq': 4953, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.843218+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:08.843218+0000', 'last_peered': '2024-09-15T21:06:08.843218+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:08.843218+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 4919, 'log_dups_size': 0, 'ondisk_log_size': 4919, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40286854, 'num_objects': 4918, 'num_object_clones': 0, 'num_object_copies': 19672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4918, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4918, 'num_write_kb': 39343, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5043", 'reported_seq': 5118, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:11.176983+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:11.176983+0000', 'last_peered': '2024-09-15T21:06:11.176983+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:11.176983+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5043, 'log_dups_size': 0, 'ondisk_log_size': 5043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41302662, 'num_objects': 5042, 'num_object_clones': 0, 'num_object_copies': 20168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5042, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5042, 'num_write_kb': 40335, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2756}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5000", 'reported_seq': 5050, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:08.753075+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:08.753075+0000', 'last_peered': '2024-09-15T21:06:08.753075+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:08.753075+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5000, 'log_dups_size': 0, 'ondisk_log_size': 5000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40924428, 'num_objects': 4996, 'num_object_clones': 0, 'num_object_copies': 19984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4996, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4998, 'num_write_kb': 39980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2717}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:12.976 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5071", 'reported_seq': 5149, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:11.380370+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:11.380370+0000', 'last_peered': '2024-09-15T21:06:11.380370+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:11.380370+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5071, 'log_dups_size': 0, 'ondisk_log_size': 5071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41514252, 'num_objects': 5068, 'num_object_clones': 0, 'num_object_copies': 20272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5069, 'num_write_kb': 40549, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2787}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:12.977 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:12.977 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5007", 'reported_seq': 5071, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:07.989377+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:07.989377+0000', 'last_peered': '2024-09-15T21:06:07.989377+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:07.989377+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5007, 'log_dups_size': 0, 'ondisk_log_size': 5007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40973580, 'num_objects': 5002, 'num_object_clones': 0, 'num_object_copies': 20008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5005, 'num_write_kb': 40028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:12.977 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:13.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 170 20 81529 81509 3.74526 2.10156 0.0907219 0.0417059 2024-09-15T21:06:13.739 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:13.739 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35546/321260 objects degraded (11.065%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":239,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":80315,"data_bytes":657901224,"bytes_used":2129899520,"bytes_avail":384417157120,"bytes_total":386547056640,"degraded_objects":35546,"degraded_total":321260,"degraded_ratio":0.11064558301687107,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":88708,"recovering_keys_per_sec":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"write_bytes_sec":2258657,"read_op_per_sec":0,"write_op_per_sec":275},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:13.758 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:13.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:14.355 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:14.355 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:14.379 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-15T21:06:13.970931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666609320,"num_objects":81378,"num_object_clones":0,"num_object_copies":325512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81405,"num_write_kb":651122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81432,"ondisk_log_size":81432,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2111340,"kb_used_data":1156396,"kb_used_omap":14,"kb_used_meta":954865,"kb_avail":375376020,"statfs":{"total":386547056640,"available":384385044480,"internally_reserved":0,"allocated":1184149504,"data_stored":1183521704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":977782322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,4,5,12,13,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32620544,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-201,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3982,"num_write_kb":31856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":1646592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005104"},"pg_stats":[{"pgid":"1.f","version":"31'5054","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.923752+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:13.923752+0000","last_peered":"2024-09-15T21:06:13.923752+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:13.923752+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5128","reported_seq":5191,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.001078+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:13.001078+0000","last_peered":"2024-09-15T21:06:13.001078+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:13.001078+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'4974","reported_seq":5048,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.354546+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:11.354546+0000","last_peered":"2024-09-15T21:06:11.354546+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:11.354546+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":4974,"log_dups_size":0,"ondisk_log_size":4974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40721030,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4973,"num_write_kb":39774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5087","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.060093+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:13.060093+0000","last_peered":"2024-09-15T21:06:13.060093+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:13.060093+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637132,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5195","reported_seq":5246,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.836855+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:13.836855+0000","last_peered":"2024-09-15T21:06:13.836855+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:13.836855+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42530060,"num_objects":5192,"num_object_clones":0,"num_object_copies":20768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":41541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5077","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.798032+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:13.798032+0000","last_peered":"2024-09-15T21:06:13.798032+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:13.798032+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5213","reported_seq":5277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.144177+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:13.144177+0000","last_peered":"2024-09-15T21:06:13.144177+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:13.144177+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5211,"num_write_kb":41668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2841}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5211","reported_seq":5275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:12.794453+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:12.794453+0000","last_peered":"2024-09-15T21:06:12.794453+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:12.794453+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42644748,"num_objects":5206,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5079","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.845410+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:13.845410+0000","last_peered":"2024-09-15T21:06:13.845410+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:13.845410+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41536024,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5079","reported_seq":5155,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.424816+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:11.424816+0000","last_peered":"2024-09-15T21:06:11.424816+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:11.424816+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41587980,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2742}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5033","reported_seq":8311,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.925652+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:13.925652+0000","last_peered":"2024-09-15T21:06:13.925652+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:13.925652+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41185170,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":5044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.835544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:13.835544+0000","last_peered":"2024-09-15T21:06:13.835544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:13.835544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024134,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5043","reported_seq":5118,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.176983+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:11.176983+0000","last_peered":"2024-09-15T21:06:11.176983+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:11.176983+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41302662,"num_objects":5042,"num_object_clones":0,"num_object_copies":20168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":40335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2756}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5086","reported_seq":5138,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.696890+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:13.696890+0000","last_peered":"2024-09-15T21:06:13.696890+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:13.696890+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2803}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5071","reported_seq":5149,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:11.380370+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:11.380370+0000","last_peered":"2024-09-15T21:06:11.380370+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:11.380370+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41514252,"num_objects":5068,"num_object_clones":0,"num_object_copies":20272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":40549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2787}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5093","reported_seq":5157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.084282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:13.084282+0000","last_peered":"2024-09-15T21:06:13.084282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:13.084282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41678092,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":666609320,"num_objects":81378,"num_object_clones":0,"num_object_copies":325512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81405,"num_write_kb":651122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186148352,"data_stored":1186148352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81432,"ondisk_log_size":81432,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582768,"kb_used_data":326176,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93789072,"statfs":{"total":96636764160,"available":96040009728,"internally_reserved":0,"allocated":334004224,"data_stored":333847274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579404,"kb_used_data":323324,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":93792436,"statfs":{"total":96636764160,"available":96043454464,"internally_reserved":0,"allocated":331083776,"data_stored":330926826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051612,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":375984,"kb_used_data":181920,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93995856,"statfs":{"total":96636764160,"available":96251756544,"internally_reserved":0,"allocated":186286080,"data_stored":186129130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573184,"kb_used_data":324976,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798656,"statfs":{"total":96636764160,"available":96049823744,"internally_reserved":0,"allocated":332775424,"data_stored":332618474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333418496,"data_stored":333418496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":186994688,"data_stored":186994688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":331522048,"data_stored":331522048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334213120,"data_stored":334213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:14.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 171 20 81796 81776 3.73555 2.08594 0.112403 0.0418135 2024-09-15T21:06:15.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 172 20 82091 82071 3.72723 2.30469 0.0549665 0.0419051 2024-09-15T21:06:16.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 173 20 82389 82369 3.71914 2.32812 0.0520727 0.0419975 2024-09-15T21:06:17.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:17.652 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 174 20 82678 82658 3.71075 2.25781 0.0774168 0.0420922 2024-09-15T21:06:17.981 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:17.982 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:18.012 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-15T21:06:17.972532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2132604,"kb_used_data":1167932,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375354756,"statfs":{"total":386547056640,"available":384363270144,"internally_reserved":0,"allocated":1195962368,"data_stored":1195334568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":987743794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,8,12,16,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26779648,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3269,"num_write_kb":26152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005089"},"pg_stats":[{"pgid":"1.f","version":"31'5054","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.923752+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:13.923752+0000","last_peered":"2024-09-15T21:06:13.923752+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:13.923752+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5128","reported_seq":5191,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.001078+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:13.001078+0000","last_peered":"2024-09-15T21:06:13.001078+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:13.001078+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.403183+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:16.403183+0000","last_peered":"2024-09-15T21:06:16.403183+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:16.403183+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2796}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5087","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.060093+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:13.060093+0000","last_peered":"2024-09-15T21:06:13.060093+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:13.060093+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637132,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5195","reported_seq":5246,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.836855+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:13.836855+0000","last_peered":"2024-09-15T21:06:13.836855+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:13.836855+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42530060,"num_objects":5192,"num_object_clones":0,"num_object_copies":20768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":41541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5077","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.798032+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:13.798032+0000","last_peered":"2024-09-15T21:06:13.798032+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:13.798032+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5213","reported_seq":5277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.144177+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:13.144177+0000","last_peered":"2024-09-15T21:06:13.144177+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:13.144177+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5211,"num_write_kb":41668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2841}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5211","reported_seq":5275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:12.794453+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:12.794453+0000","last_peered":"2024-09-15T21:06:12.794453+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:12.794453+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42644748,"num_objects":5206,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5079","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.845410+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:13.845410+0000","last_peered":"2024-09-15T21:06:13.845410+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:13.845410+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41536024,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5163","reported_seq":5239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.457696+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:16.457696+0000","last_peered":"2024-09-15T21:06:16.457696+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:16.457696+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5033","reported_seq":8311,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.925652+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:13.925652+0000","last_peered":"2024-09-15T21:06:13.925652+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:13.925652+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41185170,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":5044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.835544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:13.835544+0000","last_peered":"2024-09-15T21:06:13.835544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:13.835544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024134,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5141","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.304788+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:16.304788+0000","last_peered":"2024-09-15T21:06:16.304788+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:16.304788+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42105478,"num_objects":5140,"num_object_clones":0,"num_object_copies":20560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":41119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5086","reported_seq":5138,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.696890+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:13.696890+0000","last_peered":"2024-09-15T21:06:13.696890+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:13.696890+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2803}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5166","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.452299+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:16.452299+0000","last_peered":"2024-09-15T21:06:16.452299+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:16.452299+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292492,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":41309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2882}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5093","reported_seq":5157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.084282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:13.084282+0000","last_peered":"2024-09-15T21:06:13.084282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:13.084282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41678092,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197977600,"data_stored":1197977600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582768,"kb_used_data":326176,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93789072,"statfs":{"total":96636764160,"available":96040009728,"internally_reserved":0,"allocated":334004224,"data_stored":333847274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590592,"kb_used_data":328944,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93781248,"statfs":{"total":96636764160,"available":96031997952,"internally_reserved":0,"allocated":336838656,"data_stored":336681706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386060,"kb_used_data":187836,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93985780,"statfs":{"total":96636764160,"available":96241438720,"internally_reserved":0,"allocated":192344064,"data_stored":192187114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573184,"kb_used_data":324976,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798656,"statfs":{"total":96636764160,"available":96049823744,"internally_reserved":0,"allocated":332775424,"data_stored":332618474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333418496,"data_stored":333418496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193077248,"data_stored":193077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337268736,"data_stored":337268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334213120,"data_stored":334213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:18.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:18.567 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:18.567 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:18.593 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-15T21:06:17.972532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2132604,"kb_used_data":1167932,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375354756,"statfs":{"total":386547056640,"available":384363270144,"internally_reserved":0,"allocated":1195962368,"data_stored":1195334568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":987743794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,8,12,16,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26779648,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3269,"num_write_kb":26152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005089"},"pg_stats":[{"pgid":"1.f","version":"31'5054","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.923752+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:13.923752+0000","last_peered":"2024-09-15T21:06:13.923752+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:13.923752+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5128","reported_seq":5191,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.001078+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:13.001078+0000","last_peered":"2024-09-15T21:06:13.001078+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:13.001078+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.403183+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:16.403183+0000","last_peered":"2024-09-15T21:06:16.403183+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:16.403183+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2796}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5087","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.060093+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:13.060093+0000","last_peered":"2024-09-15T21:06:13.060093+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:13.060093+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637132,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5195","reported_seq":5246,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.836855+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:13.836855+0000","last_peered":"2024-09-15T21:06:13.836855+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:13.836855+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42530060,"num_objects":5192,"num_object_clones":0,"num_object_copies":20768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":41541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5077","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.798032+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:13.798032+0000","last_peered":"2024-09-15T21:06:13.798032+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:13.798032+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5213","reported_seq":5277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.144177+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:13.144177+0000","last_peered":"2024-09-15T21:06:13.144177+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:13.144177+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5211,"num_write_kb":41668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2841}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5211","reported_seq":5275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:12.794453+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:12.794453+0000","last_peered":"2024-09-15T21:06:12.794453+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:12.794453+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42644748,"num_objects":5206,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5079","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.845410+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:13.845410+0000","last_peered":"2024-09-15T21:06:13.845410+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:13.845410+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41536024,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5163","reported_seq":5239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.457696+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:16.457696+0000","last_peered":"2024-09-15T21:06:16.457696+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:16.457696+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5033","reported_seq":8311,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.925652+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:13.925652+0000","last_peered":"2024-09-15T21:06:13.925652+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:13.925652+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41185170,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":5044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.835544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:13.835544+0000","last_peered":"2024-09-15T21:06:13.835544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:13.835544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024134,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5141","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.304788+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:16.304788+0000","last_peered":"2024-09-15T21:06:16.304788+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:16.304788+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42105478,"num_objects":5140,"num_object_clones":0,"num_object_copies":20560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":41119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5086","reported_seq":5138,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.696890+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:13.696890+0000","last_peered":"2024-09-15T21:06:13.696890+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:13.696890+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2803}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5166","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.452299+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:16.452299+0000","last_peered":"2024-09-15T21:06:16.452299+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:16.452299+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292492,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":41309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2882}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5093","reported_seq":5157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.084282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:13.084282+0000","last_peered":"2024-09-15T21:06:13.084282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:13.084282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41678092,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197977600,"data_stored":1197977600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582768,"kb_used_data":326176,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93789072,"statfs":{"total":96636764160,"available":96040009728,"internally_reserved":0,"allocated":334004224,"data_stored":333847274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590592,"kb_used_data":328944,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93781248,"statfs":{"total":96636764160,"available":96031997952,"internally_reserved":0,"allocated":336838656,"data_stored":336681706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386060,"kb_used_data":187836,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93985780,"statfs":{"total":96636764160,"available":96241438720,"internally_reserved":0,"allocated":192344064,"data_stored":192187114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573184,"kb_used_data":324976,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798656,"statfs":{"total":96636764160,"available":96049823744,"internally_reserved":0,"allocated":332775424,"data_stored":332618474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333418496,"data_stored":333418496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193077248,"data_stored":193077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337268736,"data_stored":337268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334213120,"data_stored":334213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:18.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:18.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5054", 'reported_seq': 5095, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.923752+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:13.923752+0000', 'last_peered': '2024-09-15T21:06:13.923752+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:13.923752+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5054, 'log_dups_size': 0, 'ondisk_log_size': 5054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41402368, 'num_objects': 5054, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5054, 'num_write_kb': 40432, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2778}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:18.594 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5128", 'reported_seq': 5191, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.001078+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:13.001078+0000', 'last_peered': '2024-09-15T21:06:13.001078+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:13.001078+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5128, 'log_dups_size': 0, 'ondisk_log_size': 5128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41973004, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5126, 'num_write_kb': 41004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2744}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:18.594 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5057", 'reported_seq': 5131, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:16.403183+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:16.403183+0000', 'last_peered': '2024-09-15T21:06:16.403183+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:16.403183+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5057, 'log_dups_size': 0, 'ondisk_log_size': 5057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41400966, 'num_objects': 5054, 'num_object_clones': 0, 'num_object_copies': 20216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5056, 'num_write_kb': 40438, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2796}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5087", 'reported_seq': 5151, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.060093+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:13.060093+0000', 'last_peered': '2024-09-15T21:06:13.060093+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:13.060093+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5087, 'log_dups_size': 0, 'ondisk_log_size': 5087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41637132, 'num_objects': 5083, 'num_object_clones': 0, 'num_object_copies': 20332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5085, 'num_write_kb': 40669, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2749}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5195", 'reported_seq': 5246, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.836855+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:13.836855+0000', 'last_peered': '2024-09-15T21:06:13.836855+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:13.836855+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5195, 'log_dups_size': 0, 'ondisk_log_size': 5195, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42530060, 'num_objects': 5192, 'num_object_clones': 0, 'num_object_copies': 20768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5193, 'num_write_kb': 41541, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2881}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5077", 'reported_seq': 5117, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.798032+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:13.798032+0000', 'last_peered': '2024-09-15T21:06:13.798032+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:13.798032+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5077, 'log_dups_size': 0, 'ondisk_log_size': 5077, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41590784, 'num_objects': 5077, 'num_object_clones': 0, 'num_object_copies': 20308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5077, 'num_write_kb': 40616, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2761}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:18.595 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5213", 'reported_seq': 5277, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.144177+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:13.144177+0000', 'last_peered': '2024-09-15T21:06:13.144177+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:13.144177+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5213, 'log_dups_size': 0, 'ondisk_log_size': 5213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42652940, 'num_objects': 5207, 'num_object_clones': 0, 'num_object_copies': 20828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5211, 'num_write_kb': 41668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2841}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5211", 'reported_seq': 5275, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:12.794453+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:12.794453+0000', 'last_peered': '2024-09-15T21:06:12.794453+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:12.794453+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5211, 'log_dups_size': 0, 'ondisk_log_size': 5211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42644748, 'num_objects': 5206, 'num_object_clones': 0, 'num_object_copies': 20824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5209, 'num_write_kb': 41660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5079", 'reported_seq': 5134, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.845410+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:13.845410+0000', 'last_peered': '2024-09-15T21:06:13.845410+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:13.845410+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5079, 'log_dups_size': 0, 'ondisk_log_size': 5079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41536024, 'num_objects': 5071, 'num_object_clones': 0, 'num_object_copies': 20284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5076, 'num_write_kb': 40585, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2739}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5163", 'reported_seq': 5239, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:16.457696+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:16.457696+0000', 'last_peered': '2024-09-15T21:06:16.457696+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:16.457696+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5163, 'log_dups_size': 0, 'ondisk_log_size': 5163, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42276108, 'num_objects': 5161, 'num_object_clones': 0, 'num_object_copies': 20644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5161, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5161, 'num_write_kb': 41286, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2826}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:18.596 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5033", 'reported_seq': 8311, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.925652+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:13.925652+0000', 'last_peered': '2024-09-15T21:06:13.925652+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:13.925652+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5033, 'log_dups_size': 0, 'ondisk_log_size': 5033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41185170, 'num_objects': 5028, 'num_object_clones': 0, 'num_object_copies': 20112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 652, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5030, 'num_write_kb': 40228, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1614, 'num_bytes_recovered': 13172736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4376}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5009", 'reported_seq': 5044, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.835544+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:13.835544+0000', 'last_peered': '2024-09-15T21:06:13.835544+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:13.835544+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5009, 'log_dups_size': 0, 'ondisk_log_size': 5009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41024134, 'num_objects': 5008, 'num_object_clones': 0, 'num_object_copies': 20032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5008, 'num_write_kb': 40063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2752}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5141", 'reported_seq': 5216, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:16.304788+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:16.304788+0000', 'last_peered': '2024-09-15T21:06:16.304788+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:16.304788+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5141, 'log_dups_size': 0, 'ondisk_log_size': 5141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42105478, 'num_objects': 5140, 'num_object_clones': 0, 'num_object_copies': 20560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5140, 'num_write_kb': 41119, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5086", 'reported_seq': 5138, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.696890+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:13.696890+0000', 'last_peered': '2024-09-15T21:06:13.696890+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:13.696890+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5086, 'log_dups_size': 0, 'ondisk_log_size': 5086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41628940, 'num_objects': 5082, 'num_object_clones': 0, 'num_object_copies': 20328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5084, 'num_write_kb': 40668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5166", 'reported_seq': 5244, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:16.452299+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:16.452299+0000', 'last_peered': '2024-09-15T21:06:16.452299+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:16.452299+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5166, 'log_dups_size': 0, 'ondisk_log_size': 5166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42292492, 'num_objects': 5163, 'num_object_clones': 0, 'num_object_copies': 20652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5164, 'num_write_kb': 41309, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:18.597 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:18.598 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5093", 'reported_seq': 5157, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:13.084282+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:13.084282+0000', 'last_peered': '2024-09-15T21:06:13.084282+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:13.084282+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5093, 'log_dups_size': 0, 'ondisk_log_size': 5093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41678092, 'num_objects': 5088, 'num_object_clones': 0, 'num_object_copies': 20352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5088, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5091, 'num_write_kb': 40716, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2773}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:18.598 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:18.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 175 20 82933 82913 3.70091 1.99219 0.0972449 0.0422062 2024-09-15T21:06:19.263 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:19.264 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35475/326952 objects degraded (10.850%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":245,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81738,"data_bytes":669558440,"bytes_used":2183786496,"bytes_avail":384363270144,"bytes_total":386547056640,"degraded_objects":35475,"degraded_total":326952,"degraded_ratio":0.10850216545547971,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":92803,"recovering_keys_per_sec":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"write_bytes_sec":2230691,"read_op_per_sec":0,"write_op_per_sec":272},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:19.288 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:19.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:19.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 176 20 83188 83168 3.6912 1.99219 0.0766405 0.0423178 2024-09-15T21:06:19.899 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:19.918 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:19.941 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-15T21:06:17.972532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2132604,"kb_used_data":1167932,"kb_used_omap":14,"kb_used_meta":964593,"kb_avail":375354756,"statfs":{"total":386547056640,"available":384363270144,"internally_reserved":0,"allocated":1195962368,"data_stored":1195334568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":987743794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,1,8,12,16,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26779648,"num_objects":3269,"num_object_clones":0,"num_object_copies":13076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3269,"num_write_kb":26152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":1114112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005089"},"pg_stats":[{"pgid":"1.f","version":"31'5054","reported_seq":5095,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.923752+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:13.923752+0000","last_peered":"2024-09-15T21:06:13.923752+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:13.923752+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5054,"num_write_kb":40432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5128","reported_seq":5191,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.001078+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:13.001078+0000","last_peered":"2024-09-15T21:06:13.001078+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:13.001078+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5126,"num_write_kb":41004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2744}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5057","reported_seq":5131,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.403183+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:16.403183+0000","last_peered":"2024-09-15T21:06:16.403183+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:16.403183+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41400966,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5056,"num_write_kb":40438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2796}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5087","reported_seq":5151,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.060093+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:13.060093+0000","last_peered":"2024-09-15T21:06:13.060093+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:13.060093+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41637132,"num_objects":5083,"num_object_clones":0,"num_object_copies":20332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":40669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5195","reported_seq":5246,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.836855+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:13.836855+0000","last_peered":"2024-09-15T21:06:13.836855+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:13.836855+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5195,"log_dups_size":0,"ondisk_log_size":5195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42530060,"num_objects":5192,"num_object_clones":0,"num_object_copies":20768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5193,"num_write_kb":41541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2881}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5077","reported_seq":5117,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.798032+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:13.798032+0000","last_peered":"2024-09-15T21:06:13.798032+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:13.798032+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5077,"log_dups_size":0,"ondisk_log_size":5077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41590784,"num_objects":5077,"num_object_clones":0,"num_object_copies":20308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5077,"num_write_kb":40616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2761}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5213","reported_seq":5277,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.144177+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:13.144177+0000","last_peered":"2024-09-15T21:06:13.144177+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:13.144177+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5213,"log_dups_size":0,"ondisk_log_size":5213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5211,"num_write_kb":41668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2841}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5211","reported_seq":5275,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:12.794453+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:12.794453+0000","last_peered":"2024-09-15T21:06:12.794453+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:12.794453+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42644748,"num_objects":5206,"num_object_clones":0,"num_object_copies":20824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":41660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5079","reported_seq":5134,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.845410+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:13.845410+0000","last_peered":"2024-09-15T21:06:13.845410+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:13.845410+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41536024,"num_objects":5071,"num_object_clones":0,"num_object_copies":20284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":40585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2739}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5163","reported_seq":5239,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.457696+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:16.457696+0000","last_peered":"2024-09-15T21:06:16.457696+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:16.457696+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5163,"log_dups_size":0,"ondisk_log_size":5163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42276108,"num_objects":5161,"num_object_clones":0,"num_object_copies":20644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5161,"num_write_kb":41286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2826}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5033","reported_seq":8311,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.925652+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:13.925652+0000","last_peered":"2024-09-15T21:06:13.925652+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:13.925652+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41185170,"num_objects":5028,"num_object_clones":0,"num_object_copies":20112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":40228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4376}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5009","reported_seq":5044,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.835544+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:13.835544+0000","last_peered":"2024-09-15T21:06:13.835544+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:13.835544+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41024134,"num_objects":5008,"num_object_clones":0,"num_object_copies":20032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":40063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2752}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5141","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.304788+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:16.304788+0000","last_peered":"2024-09-15T21:06:16.304788+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:16.304788+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5141,"log_dups_size":0,"ondisk_log_size":5141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42105478,"num_objects":5140,"num_object_clones":0,"num_object_copies":20560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5140,"num_write_kb":41119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5086","reported_seq":5138,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.696890+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:13.696890+0000","last_peered":"2024-09-15T21:06:13.696890+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:13.696890+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5086,"log_dups_size":0,"ondisk_log_size":5086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41628940,"num_objects":5082,"num_object_clones":0,"num_object_copies":20328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5084,"num_write_kb":40668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2803}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5166","reported_seq":5244,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:16.452299+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:16.452299+0000","last_peered":"2024-09-15T21:06:16.452299+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:16.452299+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5166,"log_dups_size":0,"ondisk_log_size":5166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42292492,"num_objects":5163,"num_object_clones":0,"num_object_copies":20652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5164,"num_write_kb":41309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2882}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5093","reported_seq":5157,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:13.084282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:13.084282+0000","last_peered":"2024-09-15T21:06:13.084282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:13.084282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5093,"log_dups_size":0,"ondisk_log_size":5093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41678092,"num_objects":5088,"num_object_clones":0,"num_object_copies":20352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5091,"num_write_kb":40716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2773}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":669558440,"num_objects":81738,"num_object_clones":0,"num_object_copies":326952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81765,"num_write_kb":654002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1638,"num_bytes_recovered":13172736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1197977600,"data_stored":1197977600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81792,"ondisk_log_size":81792,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582768,"kb_used_data":326176,"kb_used_omap":4,"kb_used_meta":256571,"kb_avail":93789072,"statfs":{"total":96636764160,"available":96040009728,"internally_reserved":0,"allocated":334004224,"data_stored":333847274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":262729424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,2,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":590592,"kb_used_data":328944,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":93781248,"statfs":{"total":96636764160,"available":96031997952,"internally_reserved":0,"allocated":336838656,"data_stored":336681706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386060,"kb_used_data":187836,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93985780,"statfs":{"total":96636764160,"available":96241438720,"internally_reserved":0,"allocated":192344064,"data_stored":192187114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573184,"kb_used_data":324976,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93798656,"statfs":{"total":96636764160,"available":96049823744,"internally_reserved":0,"allocated":332775424,"data_stored":332618474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":254144206},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,2,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":333418496,"data_stored":333418496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193077248,"data_stored":193077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337268736,"data_stored":337268736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":334213120,"data_stored":334213120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:20.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 177 20 83472 83452 3.68288 2.21875 0.0334849 0.0424107 2024-09-15T21:06:21.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 178 20 83772 83752 3.67536 2.34375 0.0551576 0.0424987 2024-09-15T21:06:22.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 179 20 84074 84054 3.66801 2.35938 0.0628225 0.0425849 2024-09-15T21:06:22.943 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:23.528 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:23.528 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:23.562 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-15T21:06:21.973883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681256616,"num_objects":83166,"num_object_clones":0,"num_object_copies":332664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83193,"num_write_kb":665426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83220,"ondisk_log_size":83220,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2235616,"kb_used_data":1184864,"kb_used_omap":14,"kb_used_meta":1050673,"kb_avail":375251744,"statfs":{"total":386547056640,"available":384257785856,"internally_reserved":0,"allocated":1213300736,"data_stored":1212672936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1075889714},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,6,19,15,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26451968,"num_objects":3229,"num_object_clones":0,"num_object_copies":12916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3229,"num_write_kb":25832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004708"},"pg_stats":[{"pgid":"1.f","version":"31'5151","reported_seq":5192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.850811+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:18.850811+0000","last_peered":"2024-09-15T21:06:18.850811+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:18.850811+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42196992,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":41208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5204","reported_seq":5267,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.118752+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:18.118752+0000","last_peered":"2024-09-15T21:06:18.118752+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:18.118752+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5204,"log_dups_size":0,"ondisk_log_size":5204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42595596,"num_objects":5200,"num_object_clones":0,"num_object_copies":20800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5202,"num_write_kb":41612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2820}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5137","reported_seq":5211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.473275+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:21.473275+0000","last_peered":"2024-09-15T21:06:21.473275+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:21.473275+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42056326,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5177","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:17.999865+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:17.999865+0000","last_peered":"2024-09-15T21:06:17.999865+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:17.999865+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42374412,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":41389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2839}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5281","reported_seq":5332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.776047+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:18.776047+0000","last_peered":"2024-09-15T21:06:18.776047+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:18.776047+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5167","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.694715+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:18.694715+0000","last_peered":"2024-09-15T21:06:18.694715+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:18.694715+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42328064,"num_objects":5167,"num_object_clones":0,"num_object_copies":20668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":41336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5307","reported_seq":5370,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.216857+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:18.216857+0000","last_peered":"2024-09-15T21:06:18.216857+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:18.216857+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43422988,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":42420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2935}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5305","reported_seq":5369,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.163135+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:18.163135+0000","last_peered":"2024-09-15T21:06:18.163135+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:18.163135+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43414796,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2826}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5161","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.935637+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:18.935637+0000","last_peered":"2024-09-15T21:06:18.935637+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:18.935637+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42207768,"num_objects":5153,"num_object_clones":0,"num_object_copies":20612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5273","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.364194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:21.364194+0000","last_peered":"2024-09-15T21:06:21.364194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:21.364194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43177228,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5271,"num_write_kb":42166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5132","reported_seq":8558,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.918693+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:18.918693+0000","last_peered":"2024-09-15T21:06:18.918693+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:18.918693+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41996178,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5097","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.873043+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:18.873043+0000","last_peered":"2024-09-15T21:06:18.873043+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:18.873043+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41745030,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":40767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5230","reported_seq":5305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.495856+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:21.495856+0000","last_peered":"2024-09-15T21:06:21.495856+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:21.495856+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42834566,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5173","reported_seq":5224,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.926835+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:18.926835+0000","last_peered":"2024-09-15T21:06:18.926835+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:18.926835+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341644,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5247","reported_seq":5325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.529687+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:21.529687+0000","last_peered":"2024-09-15T21:06:21.529687+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:21.529687+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5178","reported_seq":5241,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.146957+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:18.146957+0000","last_peered":"2024-09-15T21:06:18.146957+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:18.146957+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42374412,"num_objects":5173,"num_object_clones":0,"num_object_copies":20692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5176,"num_write_kb":41396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681256616,"num_objects":83166,"num_object_clones":0,"num_object_copies":332664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83193,"num_write_kb":665426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1215332352,"data_stored":1215332352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83220,"ondisk_log_size":83220,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630072,"kb_used_data":331816,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93741768,"statfs":{"total":96636764160,"available":95991570432,"internally_reserved":0,"allocated":339779584,"data_stored":339622634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":305393360},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,7,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636440,"kb_used_data":334536,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93735400,"statfs":{"total":96636764160,"available":95985049600,"internally_reserved":0,"allocated":342564864,"data_stored":342407914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051613,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":386060,"kb_used_data":187836,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93985780,"statfs":{"total":96636764160,"available":96241438720,"internally_reserved":0,"allocated":192344064,"data_stored":192187114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,6,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583044,"kb_used_data":330676,"kb_used_omap":4,"kb_used_meta":252347,"kb_avail":93788796,"statfs":{"total":96636764160,"available":96039727104,"internally_reserved":0,"allocated":338612224,"data_stored":338455274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":258404046},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":339255296,"data_stored":339255296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193077248,"data_stored":193077248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343031808,"data_stored":343031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339968000,"data_stored":339968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:23.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:23.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:06:23.651421+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0426808 2024-09-15T21:06:23.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:06:23.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 180 20 84346 84326 3.65943 2.125 0.102011 0.0426808 2024-09-15T21:06:24.131 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:24.132 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:24.159 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-15T21:06:23.974686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684738216,"num_objects":83591,"num_object_clones":0,"num_object_copies":334364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83618,"num_write_kb":668826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83645,"ondisk_log_size":83645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2273596,"kb_used_data":1196476,"kb_used_omap":14,"kb_used_meta":1077041,"kb_avail":375213764,"statfs":{"total":386547056640,"available":384218894336,"internally_reserved":0,"allocated":1225191424,"data_stored":1224563624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1102890546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,3,13,12,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26836992,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004623"},"pg_stats":[{"pgid":"1.f","version":"31'5151","reported_seq":5192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.850811+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:18.850811+0000","last_peered":"2024-09-15T21:06:18.850811+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:18.850811+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42196992,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":41208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5274","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.085878+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:23.085878+0000","last_peered":"2024-09-15T21:06:23.085878+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:23.085878+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5137","reported_seq":5211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.473275+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:21.473275+0000","last_peered":"2024-09-15T21:06:21.473275+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:21.473275+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42056326,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5274","reported_seq":5338,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.155440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:23.155440+0000","last_peered":"2024-09-15T21:06:23.155440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:23.155440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5281","reported_seq":5332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.776047+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:18.776047+0000","last_peered":"2024-09-15T21:06:18.776047+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:18.776047+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5167","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.694715+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:18.694715+0000","last_peered":"2024-09-15T21:06:18.694715+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:18.694715+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42328064,"num_objects":5167,"num_object_clones":0,"num_object_copies":20668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":41336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5395","reported_seq":5457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.242329+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:23.242329+0000","last_peered":"2024-09-15T21:06:23.242329+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:23.242329+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5396","reported_seq":5460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.105847+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:23.105847+0000","last_peered":"2024-09-15T21:06:23.105847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:23.105847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160268,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5161","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.935637+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:18.935637+0000","last_peered":"2024-09-15T21:06:18.935637+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:18.935637+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42207768,"num_objects":5153,"num_object_clones":0,"num_object_copies":20612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5273","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.364194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:21.364194+0000","last_peered":"2024-09-15T21:06:21.364194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:21.364194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43177228,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5271,"num_write_kb":42166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5132","reported_seq":8558,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.918693+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:18.918693+0000","last_peered":"2024-09-15T21:06:18.918693+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:18.918693+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41996178,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5097","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.873043+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:18.873043+0000","last_peered":"2024-09-15T21:06:18.873043+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:18.873043+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41745030,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":40767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5230","reported_seq":5305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.495856+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:21.495856+0000","last_peered":"2024-09-15T21:06:21.495856+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:21.495856+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42834566,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5173","reported_seq":5224,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.926835+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:18.926835+0000","last_peered":"2024-09-15T21:06:18.926835+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:18.926835+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341644,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5247","reported_seq":5325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.529687+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:21.529687+0000","last_peered":"2024-09-15T21:06:21.529687+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:21.529687+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5257","reported_seq":5320,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.275674+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:23.275674+0000","last_peered":"2024-09-15T21:06:23.275674+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:23.275674+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021580,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684738216,"num_objects":83591,"num_object_clones":0,"num_object_copies":334364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83618,"num_write_kb":668826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1227243520,"data_stored":1227243520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83645,"ondisk_log_size":83645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630072,"kb_used_data":331816,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93741768,"statfs":{"total":96636764160,"available":95991570432,"internally_reserved":0,"allocated":339779584,"data_stored":339622634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":305393360},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,7,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636440,"kb_used_data":334536,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93735400,"statfs":{"total":96636764160,"available":95985049600,"internally_reserved":0,"allocated":342564864,"data_stored":342407914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396252,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93975588,"statfs":{"total":96636764160,"available":96231002112,"internally_reserved":0,"allocated":198520832,"data_stored":198363882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610832,"kb_used_data":336256,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93761008,"statfs":{"total":96636764160,"available":96011272192,"internally_reserved":0,"allocated":344326144,"data_stored":344169194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":345079808,"data_stored":345079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199163904,"data_stored":199163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343031808,"data_stored":343031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339968000,"data_stored":339968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:24.160 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5151", 'reported_seq': 5192, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.850811+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:18.850811+0000', 'last_peered': '2024-09-15T21:06:18.850811+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:18.850811+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5151, 'log_dups_size': 0, 'ondisk_log_size': 5151, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42196992, 'num_objects': 5151, 'num_object_clones': 0, 'num_object_copies': 20604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5151, 'num_write_kb': 41208, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2875}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5274", 'reported_seq': 5337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.085878+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:23.085878+0000', 'last_peered': '2024-09-15T21:06:23.085878+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:23.085878+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5274, 'log_dups_size': 0, 'ondisk_log_size': 5274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43169036, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5272, 'num_write_kb': 42172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5137", 'reported_seq': 5211, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:21.473275+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:21.473275+0000', 'last_peered': '2024-09-15T21:06:21.473275+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:21.473275+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5137, 'log_dups_size': 0, 'ondisk_log_size': 5137, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42056326, 'num_objects': 5134, 'num_object_clones': 0, 'num_object_copies': 20536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5136, 'num_write_kb': 41078, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2876}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5274", 'reported_seq': 5338, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.155440+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:23.155440+0000', 'last_peered': '2024-09-15T21:06:23.155440+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:23.155440+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5274, 'log_dups_size': 0, 'ondisk_log_size': 5274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43169036, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5272, 'num_write_kb': 42165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:24.161 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5281", 'reported_seq': 5332, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.776047+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:18.776047+0000', 'last_peered': '2024-09-15T21:06:18.776047+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:18.776047+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5281, 'log_dups_size': 0, 'ondisk_log_size': 5281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43234572, 'num_objects': 5278, 'num_object_clones': 0, 'num_object_copies': 21112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5279, 'num_write_kb': 42229, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2967}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5167", 'reported_seq': 5207, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.694715+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:18.694715+0000', 'last_peered': '2024-09-15T21:06:18.694715+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:18.694715+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5167, 'log_dups_size': 0, 'ondisk_log_size': 5167, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42328064, 'num_objects': 5167, 'num_object_clones': 0, 'num_object_copies': 20668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5167, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5167, 'num_write_kb': 41336, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2851}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5395", 'reported_seq': 5457, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.242329+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:23.242329+0000', 'last_peered': '2024-09-15T21:06:23.242329+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:23.242329+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5395, 'log_dups_size': 0, 'ondisk_log_size': 5395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44143884, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5393, 'num_write_kb': 43124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3023}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5396", 'reported_seq': 5460, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.105847+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:23.105847+0000', 'last_peered': '2024-09-15T21:06:23.105847+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:23.105847+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5396, 'log_dups_size': 0, 'ondisk_log_size': 5396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44160268, 'num_objects': 5391, 'num_object_clones': 0, 'num_object_copies': 21564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5394, 'num_write_kb': 43140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5161", 'reported_seq': 5216, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.935637+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:18.935637+0000', 'last_peered': '2024-09-15T21:06:18.935637+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:18.935637+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5161, 'log_dups_size': 0, 'ondisk_log_size': 5161, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42207768, 'num_objects': 5153, 'num_object_clones': 0, 'num_object_copies': 20612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5158, 'num_write_kb': 41241, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5273", 'reported_seq': 5349, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:21.364194+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:21.364194+0000', 'last_peered': '2024-09-15T21:06:21.364194+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:21.364194+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5273, 'log_dups_size': 0, 'ondisk_log_size': 5273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43177228, 'num_objects': 5271, 'num_object_clones': 0, 'num_object_copies': 21084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5271, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5271, 'num_write_kb': 42166, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2936}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:24.162 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5132", 'reported_seq': 8558, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.918693+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:18.918693+0000', 'last_peered': '2024-09-15T21:06:18.918693+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:18.918693+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5132, 'log_dups_size': 0, 'ondisk_log_size': 5132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41996178, 'num_objects': 5127, 'num_object_clones': 0, 'num_object_copies': 20508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 578, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5127, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5129, 'num_write_kb': 41020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1688, 'num_bytes_recovered': 13778944, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4549}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5097", 'reported_seq': 5132, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.873043+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:18.873043+0000', 'last_peered': '2024-09-15T21:06:18.873043+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:18.873043+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5097, 'log_dups_size': 0, 'ondisk_log_size': 5097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41745030, 'num_objects': 5096, 'num_object_clones': 0, 'num_object_copies': 20384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5096, 'num_write_kb': 40767, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5230", 'reported_seq': 5305, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:21.495856+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:21.495856+0000', 'last_peered': '2024-09-15T21:06:21.495856+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:21.495856+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5230, 'log_dups_size': 0, 'ondisk_log_size': 5230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42834566, 'num_objects': 5229, 'num_object_clones': 0, 'num_object_copies': 20916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5229, 'num_write_kb': 41831, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5173", 'reported_seq': 5224, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:18.926835+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:18.926835+0000', 'last_peered': '2024-09-15T21:06:18.926835+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:18.926835+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5173, 'log_dups_size': 0, 'ondisk_log_size': 5173, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42341644, 'num_objects': 5169, 'num_object_clones': 0, 'num_object_copies': 20676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5169, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5171, 'num_write_kb': 41364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5247", 'reported_seq': 5325, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:21.529687+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:21.529687+0000', 'last_peered': '2024-09-15T21:06:21.529687+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:21.529687+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5247, 'log_dups_size': 0, 'ondisk_log_size': 5247, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42956044, 'num_objects': 5244, 'num_object_clones': 0, 'num_object_copies': 20976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5244, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5245, 'num_write_kb': 41957, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:24.163 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5257", 'reported_seq': 5320, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.275674+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:23.275674+0000', 'last_peered': '2024-09-15T21:06:23.275674+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:23.275674+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5257, 'log_dups_size': 0, 'ondisk_log_size': 5257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43021580, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 21008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5255, 'num_write_kb': 42028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:24.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:24.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 181 20 84615 84595 3.65083 2.10156 0.0452319 0.0427846 2024-09-15T21:06:24.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:24.874 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35401/334364 objects degraded (10.588%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":250,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83591,"data_bytes":684738216,"bytes_used":2328162304,"bytes_avail":384218894336,"bytes_total":386547056640,"degraded_objects":35401,"degraded_total":334364,"degraded_ratio":0.10587563254417341,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":98948,"recovering_keys_per_sec":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"write_bytes_sec":2235554,"read_op_per_sec":0,"write_op_per_sec":272},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:24.906 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:24.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:25.498 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:25.498 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:25.526 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-15T21:06:23.974686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":684738216,"num_objects":83591,"num_object_clones":0,"num_object_copies":334364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83618,"num_write_kb":668826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83645,"ondisk_log_size":83645,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2273596,"kb_used_data":1196476,"kb_used_omap":14,"kb_used_meta":1077041,"kb_avail":375213764,"statfs":{"total":386547056640,"available":384218894336,"internally_reserved":0,"allocated":1225191424,"data_stored":1224563624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1102890546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,3,13,12,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26836992,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-145,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":26208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":1187840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004623"},"pg_stats":[{"pgid":"1.f","version":"31'5151","reported_seq":5192,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.850811+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:18.850811+0000","last_peered":"2024-09-15T21:06:18.850811+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:18.850811+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42196992,"num_objects":5151,"num_object_clones":0,"num_object_copies":20604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":41208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5274","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.085878+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:23.085878+0000","last_peered":"2024-09-15T21:06:23.085878+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:23.085878+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5137","reported_seq":5211,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.473275+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:21.473275+0000","last_peered":"2024-09-15T21:06:21.473275+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:21.473275+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5137,"log_dups_size":0,"ondisk_log_size":5137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42056326,"num_objects":5134,"num_object_clones":0,"num_object_copies":20536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5136,"num_write_kb":41078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2876}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5274","reported_seq":5338,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.155440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:23.155440+0000","last_peered":"2024-09-15T21:06:23.155440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:23.155440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5281","reported_seq":5332,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.776047+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:18.776047+0000","last_peered":"2024-09-15T21:06:18.776047+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:18.776047+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5281,"log_dups_size":0,"ondisk_log_size":5281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5279,"num_write_kb":42229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2967}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5167","reported_seq":5207,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.694715+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:18.694715+0000","last_peered":"2024-09-15T21:06:18.694715+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:18.694715+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42328064,"num_objects":5167,"num_object_clones":0,"num_object_copies":20668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":41336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5395","reported_seq":5457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.242329+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:23.242329+0000","last_peered":"2024-09-15T21:06:23.242329+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:23.242329+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5396","reported_seq":5460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.105847+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:23.105847+0000","last_peered":"2024-09-15T21:06:23.105847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:23.105847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160268,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5161","reported_seq":5216,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.935637+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:18.935637+0000","last_peered":"2024-09-15T21:06:18.935637+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:18.935637+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5161,"log_dups_size":0,"ondisk_log_size":5161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42207768,"num_objects":5153,"num_object_clones":0,"num_object_copies":20612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":41241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5273","reported_seq":5349,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.364194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:21.364194+0000","last_peered":"2024-09-15T21:06:21.364194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:21.364194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5273,"log_dups_size":0,"ondisk_log_size":5273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43177228,"num_objects":5271,"num_object_clones":0,"num_object_copies":21084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5271,"num_write_kb":42166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2936}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5132","reported_seq":8558,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.918693+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:18.918693+0000","last_peered":"2024-09-15T21:06:18.918693+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:18.918693+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41996178,"num_objects":5127,"num_object_clones":0,"num_object_copies":20508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5129,"num_write_kb":41020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1688,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4549}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5097","reported_seq":5132,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.873043+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:18.873043+0000","last_peered":"2024-09-15T21:06:18.873043+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:18.873043+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41745030,"num_objects":5096,"num_object_clones":0,"num_object_copies":20384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":40767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5230","reported_seq":5305,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.495856+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:21.495856+0000","last_peered":"2024-09-15T21:06:21.495856+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:21.495856+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42834566,"num_objects":5229,"num_object_clones":0,"num_object_copies":20916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2943}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5173","reported_seq":5224,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:18.926835+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:18.926835+0000","last_peered":"2024-09-15T21:06:18.926835+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:18.926835+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42341644,"num_objects":5169,"num_object_clones":0,"num_object_copies":20676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":41364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5247","reported_seq":5325,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:21.529687+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:21.529687+0000","last_peered":"2024-09-15T21:06:21.529687+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:21.529687+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42956044,"num_objects":5244,"num_object_clones":0,"num_object_copies":20976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5245,"num_write_kb":41957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5257","reported_seq":5320,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.275674+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:23.275674+0000","last_peered":"2024-09-15T21:06:23.275674+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:23.275674+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021580,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":684738216,"num_objects":83591,"num_object_clones":0,"num_object_copies":334364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83618,"num_write_kb":668826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":13778944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1227243520,"data_stored":1227243520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83645,"ondisk_log_size":83645,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":630072,"kb_used_data":331816,"kb_used_omap":4,"kb_used_meta":298235,"kb_avail":93741768,"statfs":{"total":96636764160,"available":95991570432,"internally_reserved":0,"allocated":339779584,"data_stored":339622634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":305393360},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,6,0,7,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":636440,"kb_used_data":334536,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93735400,"statfs":{"total":96636764160,"available":95985049600,"internally_reserved":0,"allocated":342564864,"data_stored":342407914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051614,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":396252,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93975588,"statfs":{"total":96636764160,"available":96231002112,"internally_reserved":0,"allocated":198520832,"data_stored":198363882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610832,"kb_used_data":336256,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93761008,"statfs":{"total":96636764160,"available":96011272192,"internally_reserved":0,"allocated":344326144,"data_stored":344169194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":345079808,"data_stored":345079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199163904,"data_stored":199163904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":343031808,"data_stored":343031808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":339968000,"data_stored":339968000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:25.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 182 20 84882 84862 3.64223 2.08594 0.104612 0.0428863 2024-09-15T21:06:26.653 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 183 20 85170 85150 3.63462 2.25 0.169082 0.0429753 2024-09-15T21:06:27.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 184 20 85470 85450 3.62755 2.34375 0.0855268 0.0430635 2024-09-15T21:06:28.528 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:28.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 185 20 85760 85740 3.62019 2.26562 0.0636645 0.0431436 2024-09-15T21:06:29.140 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:29.140 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:29.162 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-15T21:06:27.975934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692946600,"num_objects":84593,"num_object_clones":0,"num_object_copies":338372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84620,"num_write_kb":676842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84647,"ondisk_log_size":84647,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2321336,"kb_used_data":1213688,"kb_used_omap":14,"kb_used_meta":1107569,"kb_avail":375166024,"statfs":{"total":386547056640,"available":384170008576,"internally_reserved":0,"allocated":1242816512,"data_stored":1242188712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1134151218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,13,7,19,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26337280,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"1.f","version":"31'5237","reported_seq":5278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.936259+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:23.936259+0000","last_peered":"2024-09-15T21:06:23.936259+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:23.936259+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42901504,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5274","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.085878+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:23.085878+0000","last_peered":"2024-09-15T21:06:23.085878+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:23.085878+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5222","reported_seq":5296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.514558+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:26.514558+0000","last_peered":"2024-09-15T21:06:26.514558+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:26.514558+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42752646,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5274","reported_seq":5338,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.155440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:23.155440+0000","last_peered":"2024-09-15T21:06:23.155440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:23.155440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5378","reported_seq":5429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.706505+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:23.706505+0000","last_peered":"2024-09-15T21:06:23.706505+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:23.706505+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44029196,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":43005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5256","reported_seq":5295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.886762+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:23.886762+0000","last_peered":"2024-09-15T21:06:23.886762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:23.886762+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":5256,"num_object_clones":0,"num_object_copies":21024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5395","reported_seq":5457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.242329+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:23.242329+0000","last_peered":"2024-09-15T21:06:23.242329+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:23.242329+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5396","reported_seq":5460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.105847+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:23.105847+0000","last_peered":"2024-09-15T21:06:23.105847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:23.105847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160268,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5240","reported_seq":5295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.758199+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:23.758199+0000","last_peered":"2024-09-15T21:06:23.758199+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:23.758199+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42854936,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.577008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:26.577008+0000","last_peered":"2024-09-15T21:06:26.577008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:26.577008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889932,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5232","reported_seq":8806,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.859927+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:23.859927+0000","last_peered":"2024-09-15T21:06:23.859927+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:23.859927+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42815378,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5196","reported_seq":5231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.974024+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:23.974024+0000","last_peered":"2024-09-15T21:06:23.974024+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:23.974024+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556038,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5329","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.488301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:26.488301+0000","last_peered":"2024-09-15T21:06:26.488301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:26.488301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43645574,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5261","reported_seq":5313,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.896462+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:23.896462+0000","last_peered":"2024-09-15T21:06:23.896462+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:23.896462+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43062540,"num_objects":5257,"num_object_clones":0,"num_object_copies":21028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5340","reported_seq":5418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.503762+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:26.503762+0000","last_peered":"2024-09-15T21:06:26.503762+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:26.503762+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717900,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5257","reported_seq":5320,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.275674+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:23.275674+0000","last_peered":"2024-09-15T21:06:23.275674+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:23.275674+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021580,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692946600,"num_objects":84593,"num_object_clones":0,"num_object_copies":338372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84620,"num_write_kb":676842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1245020160,"data_stored":1245020160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84647,"ondisk_log_size":84647,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639892,"kb_used_data":337476,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93731948,"statfs":{"total":96636764160,"available":95981514752,"internally_reserved":0,"allocated":345575424,"data_stored":345418474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":309653200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664320,"kb_used_data":340208,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93707520,"statfs":{"total":96636764160,"available":95956500480,"internally_reserved":0,"allocated":348372992,"data_stored":348216042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406292,"kb_used_data":199748,"kb_used_omap":4,"kb_used_meta":206523,"kb_avail":93965548,"statfs":{"total":96636764160,"available":96220721152,"internally_reserved":0,"allocated":204541952,"data_stored":204385002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":211480271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610832,"kb_used_data":336256,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93761008,"statfs":{"total":96636764160,"available":96011272192,"internally_reserved":0,"allocated":344326144,"data_stored":344169194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":345079808,"data_stored":345079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205250560,"data_stored":205250560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348827648,"data_stored":348827648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345862144,"data_stored":345862144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:29.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:29.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 186 20 86018 85998 3.61156 2.01562 0.0577474 0.0432515 2024-09-15T21:06:29.754 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:29.754 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:29.778 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-15T21:06:27.975934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692946600,"num_objects":84593,"num_object_clones":0,"num_object_copies":338372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84620,"num_write_kb":676842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84647,"ondisk_log_size":84647,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2321336,"kb_used_data":1213688,"kb_used_omap":14,"kb_used_meta":1107569,"kb_avail":375166024,"statfs":{"total":386547056640,"available":384170008576,"internally_reserved":0,"allocated":1242816512,"data_stored":1242188712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1134151218},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,13,7,19,12,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26337280,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":25720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1212416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004179"},"pg_stats":[{"pgid":"1.f","version":"31'5237","reported_seq":5278,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.936259+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:23.936259+0000","last_peered":"2024-09-15T21:06:23.936259+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:23.936259+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5237,"log_dups_size":0,"ondisk_log_size":5237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42901504,"num_objects":5237,"num_object_clones":0,"num_object_copies":20948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5274","reported_seq":5337,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.085878+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:23.085878+0000","last_peered":"2024-09-15T21:06:23.085878+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:23.085878+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2890}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5222","reported_seq":5296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.514558+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:26.514558+0000","last_peered":"2024-09-15T21:06:26.514558+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:26.514558+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42752646,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5274","reported_seq":5338,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.155440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:23.155440+0000","last_peered":"2024-09-15T21:06:23.155440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:23.155440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43169036,"num_objects":5270,"num_object_clones":0,"num_object_copies":21080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5272,"num_write_kb":42165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2936}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5378","reported_seq":5429,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.706505+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:23.706505+0000","last_peered":"2024-09-15T21:06:23.706505+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:23.706505+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5378,"log_dups_size":0,"ondisk_log_size":5378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44029196,"num_objects":5375,"num_object_clones":0,"num_object_copies":21500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5376,"num_write_kb":43005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5256","reported_seq":5295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.886762+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:23.886762+0000","last_peered":"2024-09-15T21:06:23.886762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:23.886762+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":5256,"num_object_clones":0,"num_object_copies":21024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5395","reported_seq":5457,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.242329+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:23.242329+0000","last_peered":"2024-09-15T21:06:23.242329+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:23.242329+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44143884,"num_objects":5389,"num_object_clones":0,"num_object_copies":21556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5393,"num_write_kb":43124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3023}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5396","reported_seq":5460,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.105847+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:23.105847+0000","last_peered":"2024-09-15T21:06:23.105847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:23.105847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44160268,"num_objects":5391,"num_object_clones":0,"num_object_copies":21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2917}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5240","reported_seq":5295,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.758199+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:23.758199+0000","last_peered":"2024-09-15T21:06:23.758199+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:23.758199+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5240,"log_dups_size":0,"ondisk_log_size":5240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42854936,"num_objects":5232,"num_object_clones":0,"num_object_copies":20928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5237,"num_write_kb":41873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.577008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:26.577008+0000","last_peered":"2024-09-15T21:06:26.577008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:26.577008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889932,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5232","reported_seq":8806,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.859927+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:23.859927+0000","last_peered":"2024-09-15T21:06:23.859927+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:23.859927+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5232,"log_dups_size":0,"ondisk_log_size":5232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42815378,"num_objects":5227,"num_object_clones":0,"num_object_copies":20908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5229,"num_write_kb":41820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1762,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4723}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5196","reported_seq":5231,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.974024+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:23.974024+0000","last_peered":"2024-09-15T21:06:23.974024+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:23.974024+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556038,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5195,"num_write_kb":41559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5329","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.488301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:26.488301+0000","last_peered":"2024-09-15T21:06:26.488301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:26.488301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43645574,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5261","reported_seq":5313,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.896462+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:23.896462+0000","last_peered":"2024-09-15T21:06:23.896462+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:23.896462+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43062540,"num_objects":5257,"num_object_clones":0,"num_object_copies":21028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5259,"num_write_kb":42068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2978}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5340","reported_seq":5418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.503762+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:26.503762+0000","last_peered":"2024-09-15T21:06:26.503762+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:26.503762+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717900,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5257","reported_seq":5320,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:23.275674+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:23.275674+0000","last_peered":"2024-09-15T21:06:23.275674+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:23.275674+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5257,"log_dups_size":0,"ondisk_log_size":5257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43021580,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5255,"num_write_kb":42028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2937}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692946600,"num_objects":84593,"num_object_clones":0,"num_object_copies":338372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":84620,"num_write_kb":676842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1786,"num_bytes_recovered":14385152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1245020160,"data_stored":1245020160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84647,"ondisk_log_size":84647,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":639892,"kb_used_data":337476,"kb_used_omap":4,"kb_used_meta":302395,"kb_avail":93731948,"statfs":{"total":96636764160,"available":95981514752,"internally_reserved":0,"allocated":345575424,"data_stored":345418474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":309653200},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664320,"kb_used_data":340208,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93707520,"statfs":{"total":96636764160,"available":95956500480,"internally_reserved":0,"allocated":348372992,"data_stored":348216042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406292,"kb_used_data":199748,"kb_used_omap":4,"kb_used_meta":206523,"kb_avail":93965548,"statfs":{"total":96636764160,"available":96220721152,"internally_reserved":0,"allocated":204541952,"data_stored":204385002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":211480271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":610832,"kb_used_data":336256,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93761008,"statfs":{"total":96636764160,"available":96011272192,"internally_reserved":0,"allocated":344326144,"data_stored":344169194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":281145038},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":345079808,"data_stored":345079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205250560,"data_stored":205250560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348827648,"data_stored":348827648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":345862144,"data_stored":345862144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:29.779 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:29.779 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5237", 'reported_seq': 5278, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.936259+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:23.936259+0000', 'last_peered': '2024-09-15T21:06:23.936259+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:23.936259+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5237, 'log_dups_size': 0, 'ondisk_log_size': 5237, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42901504, 'num_objects': 5237, 'num_object_clones': 0, 'num_object_copies': 20948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5237, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5237, 'num_write_kb': 41896, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2961}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5274", 'reported_seq': 5337, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.085878+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:23.085878+0000', 'last_peered': '2024-09-15T21:06:23.085878+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:23.085878+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5274, 'log_dups_size': 0, 'ondisk_log_size': 5274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43169036, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5272, 'num_write_kb': 42172, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5222", 'reported_seq': 5296, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:26.514558+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:26.514558+0000', 'last_peered': '2024-09-15T21:06:26.514558+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:26.514558+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5222, 'log_dups_size': 0, 'ondisk_log_size': 5222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42752646, 'num_objects': 5219, 'num_object_clones': 0, 'num_object_copies': 20876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5219, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5221, 'num_write_kb': 41758, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2961}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5274", 'reported_seq': 5338, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.155440+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:23.155440+0000', 'last_peered': '2024-09-15T21:06:23.155440+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:23.155440+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5274, 'log_dups_size': 0, 'ondisk_log_size': 5274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43169036, 'num_objects': 5270, 'num_object_clones': 0, 'num_object_copies': 21080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5270, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5272, 'num_write_kb': 42165, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2936}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5378", 'reported_seq': 5429, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.706505+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:23.706505+0000', 'last_peered': '2024-09-15T21:06:23.706505+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:23.706505+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5378, 'log_dups_size': 0, 'ondisk_log_size': 5378, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44029196, 'num_objects': 5375, 'num_object_clones': 0, 'num_object_copies': 21500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5375, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5376, 'num_write_kb': 43005, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:29.780 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5256", 'reported_seq': 5295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.886762+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:23.886762+0000', 'last_peered': '2024-09-15T21:06:23.886762+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:23.886762+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5256, 'log_dups_size': 0, 'ondisk_log_size': 5256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43057152, 'num_objects': 5256, 'num_object_clones': 0, 'num_object_copies': 21024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5256, 'num_write_kb': 42048, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2940}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5395", 'reported_seq': 5457, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.242329+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:23.242329+0000', 'last_peered': '2024-09-15T21:06:23.242329+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:23.242329+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5395, 'log_dups_size': 0, 'ondisk_log_size': 5395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44143884, 'num_objects': 5389, 'num_object_clones': 0, 'num_object_copies': 21556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5389, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5393, 'num_write_kb': 43124, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3023}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5396", 'reported_seq': 5460, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.105847+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:23.105847+0000', 'last_peered': '2024-09-15T21:06:23.105847+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:23.105847+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5396, 'log_dups_size': 0, 'ondisk_log_size': 5396, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44160268, 'num_objects': 5391, 'num_object_clones': 0, 'num_object_copies': 21564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5394, 'num_write_kb': 43140, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2917}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5240", 'reported_seq': 5295, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.758199+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:23.758199+0000', 'last_peered': '2024-09-15T21:06:23.758199+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:23.758199+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5240, 'log_dups_size': 0, 'ondisk_log_size': 5240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42854936, 'num_objects': 5232, 'num_object_clones': 0, 'num_object_copies': 20928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5237, 'num_write_kb': 41873, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2900}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5360", 'reported_seq': 5436, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:26.577008+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:26.577008+0000', 'last_peered': '2024-09-15T21:06:26.577008+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:26.577008+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5360, 'log_dups_size': 0, 'ondisk_log_size': 5360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43889932, 'num_objects': 5358, 'num_object_clones': 0, 'num_object_copies': 21432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5358, 'num_write_kb': 42862, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3023}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:29.781 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5232", 'reported_seq': 8806, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.859927+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:23.859927+0000', 'last_peered': '2024-09-15T21:06:23.859927+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:23.859927+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5232, 'log_dups_size': 0, 'ondisk_log_size': 5232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42815378, 'num_objects': 5227, 'num_object_clones': 0, 'num_object_copies': 20908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 504, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5229, 'num_write_kb': 41820, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1762, 'num_bytes_recovered': 14385152, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4723}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5196", 'reported_seq': 5231, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.974024+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:23.974024+0000', 'last_peered': '2024-09-15T21:06:23.974024+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:23.974024+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5196, 'log_dups_size': 0, 'ondisk_log_size': 5196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42556038, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 20780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5195, 'num_write_kb': 41559, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5329", 'reported_seq': 5404, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:26.488301+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:26.488301+0000', 'last_peered': '2024-09-15T21:06:26.488301+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:26.488301+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5329, 'log_dups_size': 0, 'ondisk_log_size': 5329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43645574, 'num_objects': 5328, 'num_object_clones': 0, 'num_object_copies': 21312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5328, 'num_write_kb': 42623, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3042}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5261", 'reported_seq': 5313, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.896462+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:23.896462+0000', 'last_peered': '2024-09-15T21:06:23.896462+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:23.896462+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5261, 'log_dups_size': 0, 'ondisk_log_size': 5261, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43062540, 'num_objects': 5257, 'num_object_clones': 0, 'num_object_copies': 21028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5259, 'num_write_kb': 42068, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5340", 'reported_seq': 5418, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:26.503762+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:26.503762+0000', 'last_peered': '2024-09-15T21:06:26.503762+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:26.503762+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5340, 'log_dups_size': 0, 'ondisk_log_size': 5340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43717900, 'num_objects': 5337, 'num_object_clones': 0, 'num_object_copies': 21348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5337, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5338, 'num_write_kb': 42701, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3056}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:29.782 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5257", 'reported_seq': 5320, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:23.275674+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:23.275674+0000', 'last_peered': '2024-09-15T21:06:23.275674+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:23.275674+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5257, 'log_dups_size': 0, 'ondisk_log_size': 5257, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43021580, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 21008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5255, 'num_write_kb': 42028, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:29.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:30.545 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:30.545 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35257/342628 objects degraded (10.290%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":256,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85657,"data_bytes":701662888,"bytes_used":2417086464,"bytes_avail":384129970176,"bytes_total":386547056640,"degraded_objects":35257,"degraded_total":342628,"degraded_ratio":0.10290168929567928,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":148770,"recovering_keys_per_sec":0,"num_objects_recovered":218,"num_bytes_recovered":1785856,"num_keys_recovered":0,"write_bytes_sec":2674457,"read_op_per_sec":0,"write_op_per_sec":326},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:30.571 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:30.572 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:30.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 187 20 86270 86250 3.60278 1.96875 0.0749256 0.0433542 2024-09-15T21:06:31.171 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:31.171 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:31.192 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-15T21:06:29.976629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":701662888,"num_objects":85657,"num_object_clones":0,"num_object_copies":342628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85684,"num_write_kb":685354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85711,"ondisk_log_size":85711,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2360436,"kb_used_data":1225012,"kb_used_omap":14,"kb_used_meta":1135345,"kb_avail":375126924,"statfs":{"total":386547056640,"available":384129970176,"internally_reserved":0,"allocated":1254412288,"data_stored":1253784488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1162593842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,5,15,14,19,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32104448,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3919,"num_write_kb":31352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":1785856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004096"},"pg_stats":[{"pgid":"1.f","version":"31'5306","reported_seq":5345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.871641+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:28.871641+0000","last_peered":"2024-09-15T21:06:28.871641+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:28.871641+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5370","reported_seq":5433,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.293154+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:28.293154+0000","last_peered":"2024-09-15T21:06:28.293154+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:28.293154+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43955468,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5368,"num_write_kb":42940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":2986}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5222","reported_seq":5296,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.514558+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:26.514558+0000","last_peered":"2024-09-15T21:06:26.514558+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:26.514558+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42752646,"num_objects":5219,"num_object_clones":0,"num_object_copies":20876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5221,"num_write_kb":41758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2961}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5368","reported_seq":5432,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.146374+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:28.146374+0000","last_peered":"2024-09-15T21:06:28.146374+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:28.146374+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43939084,"num_objects":5364,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5366,"num_write_kb":42917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3030}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5448","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.942840+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:28.942840+0000","last_peered":"2024-09-15T21:06:28.942840+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:28.942840+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602636,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5345","reported_seq":5385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.049370+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:29.049370+0000","last_peered":"2024-09-15T21:06:29.049370+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:29.049370+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43786240,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5497","reported_seq":5562,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.288968+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:28.288968+0000","last_peered":"2024-09-15T21:06:28.288968+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:28.288968+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44979468,"num_objects":5491,"num_object_clones":0,"num_object_copies":21964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5495,"num_write_kb":43940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3125}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5477","reported_seq":5541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.260594+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:28.260594+0000","last_peered":"2024-09-15T21:06:28.260594+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:28.260594+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5477,"log_dups_size":0,"ondisk_log_size":5477,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44823820,"num_objects":5472,"num_object_clones":0,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2998}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5337","reported_seq":5391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.041538+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:29.041538+0000","last_peered":"2024-09-15T21:06:29.041538+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:29.041538+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43649560,"num_objects":5329,"num_object_clones":0,"num_object_copies":21316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5334,"num_write_kb":42649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5360","reported_seq":5436,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.577008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:26.577008+0000","last_peered":"2024-09-15T21:06:26.577008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:26.577008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43889932,"num_objects":5358,"num_object_clones":0,"num_object_copies":21432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5358,"num_write_kb":42862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3023}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5322","reported_seq":9036,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.066248+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:29.066248+0000","last_peered":"2024-09-15T21:06:29.066248+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:29.066248+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5322,"log_dups_size":0,"ondisk_log_size":5322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43552658,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1832,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5302","reported_seq":5336,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.065188+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:29.065188+0000","last_peered":"2024-09-15T21:06:29.065188+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:29.065188+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5329","reported_seq":5404,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.488301+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:26.488301+0000","last_peered":"2024-09-15T21:06:26.488301+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:26.488301+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43645574,"num_objects":5328,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5328,"num_write_kb":42623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3042}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5345","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.005409+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:29.005409+0000","last_peered":"2024-09-15T21:06:29.005409+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:29.005409+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43750668,"num_objects":5341,"num_object_clones":0,"num_object_copies":21364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5340","reported_seq":5418,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:26.503762+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:26.503762+0000","last_peered":"2024-09-15T21:06:26.503762+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:26.503762+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5340,"log_dups_size":0,"ondisk_log_size":5340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43717900,"num_objects":5337,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5338,"num_write_kb":42701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5343","reported_seq":5407,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.122792+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:28.122792+0000","last_peered":"2024-09-15T21:06:28.122792+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:28.122792+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5343,"log_dups_size":0,"ondisk_log_size":5343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43726092,"num_objects":5338,"num_object_clones":0,"num_object_copies":21352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5341,"num_write_kb":42716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":701662888,"num_objects":85657,"num_object_clones":0,"num_object_copies":342628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85684,"num_write_kb":685354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1256644608,"data_stored":1256644608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85711,"ondisk_log_size":85711,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667752,"kb_used_data":343128,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93704088,"statfs":{"total":96636764160,"available":95952986112,"internally_reserved":0,"allocated":351363072,"data_stored":351206122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":664320,"kb_used_data":340208,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93707520,"statfs":{"total":96636764160,"available":95956500480,"internally_reserved":0,"allocated":348372992,"data_stored":348216042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,2,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":406292,"kb_used_data":199748,"kb_used_omap":4,"kb_used_meta":206523,"kb_avail":93965548,"statfs":{"total":96636764160,"available":96220721152,"internally_reserved":0,"allocated":204541952,"data_stored":204385002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":211480271},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":622072,"kb_used_data":341928,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93749768,"statfs":{"total":96636764160,"available":95999762432,"internally_reserved":0,"allocated":350134272,"data_stored":349977322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":286846670},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,1,4,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350867456,"data_stored":350867456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":205250560,"data_stored":205250560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":348827648,"data_stored":348827648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351698944,"data_stored":351698944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:31.656 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 188 20 86543 86523 3.59496 2.13281 0.0453557 0.0434454 2024-09-15T21:06:32.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 189 20 86832 86812 3.58788 2.25781 0.0592768 0.0435351 2024-09-15T21:06:33.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 190 20 87128 87108 3.58117 2.3125 0.0394954 0.0436179 2024-09-15T21:06:34.194 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:34.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 191 20 87400 87380 3.57354 2.125 0.164612 0.0437067 2024-09-15T21:06:34.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:34.782 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:34.803 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-15T21:06:33.978259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":708011688,"num_objects":86432,"num_object_clones":0,"num_object_copies":345728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86459,"num_write_kb":691554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86486,"ondisk_log_size":86486,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2373080,"kb_used_data":1241808,"kb_used_omap":14,"kb_used_meta":1131185,"kb_avail":375114280,"statfs":{"total":386547056640,"available":384117022720,"internally_reserved":0,"allocated":1271611392,"data_stored":1270983592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1158334002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,14,18,7,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26755072,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004376"},"pg_stats":[{"pgid":"1.f","version":"31'5306","reported_seq":5345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.871641+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:28.871641+0000","last_peered":"2024-09-15T21:06:28.871641+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:28.871641+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5470","reported_seq":5533,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.300577+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:33.300577+0000","last_peered":"2024-09-15T21:06:33.300577+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:33.300577+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":43740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5304","reported_seq":5378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.605541+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:31.605541+0000","last_peered":"2024-09-15T21:06:31.605541+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:31.605541+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5444","reported_seq":5508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.266202+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:33.266202+0000","last_peered":"2024-09-15T21:06:33.266202+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:33.266202+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5448","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.942840+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:28.942840+0000","last_peered":"2024-09-15T21:06:28.942840+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:28.942840+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602636,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5345","reported_seq":5385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.049370+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:29.049370+0000","last_peered":"2024-09-15T21:06:29.049370+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:29.049370+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43786240,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5581","reported_seq":5645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.243495+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:33.243495+0000","last_peered":"2024-09-15T21:06:33.243495+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:33.243495+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667596,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5569","reported_seq":5633,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.254844+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:33.254844+0000","last_peered":"2024-09-15T21:06:33.254844+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:33.254844+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577484,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5337","reported_seq":5391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.041538+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:29.041538+0000","last_peered":"2024-09-15T21:06:29.041538+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:29.041538+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43649560,"num_objects":5329,"num_object_clones":0,"num_object_copies":21316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5334,"num_write_kb":42649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5442","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.469345+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:31.469345+0000","last_peered":"2024-09-15T21:06:31.469345+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:31.469345+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3105}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5322","reported_seq":9036,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.066248+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:29.066248+0000","last_peered":"2024-09-15T21:06:29.066248+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:29.066248+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5322,"log_dups_size":0,"ondisk_log_size":5322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43552658,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1832,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5302","reported_seq":5336,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.065188+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:29.065188+0000","last_peered":"2024-09-15T21:06:29.065188+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:29.065188+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5425","reported_seq":5500,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.633046+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:31.633046+0000","last_peered":"2024-09-15T21:06:31.633046+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:31.633046+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44432006,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5345","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.005409+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:29.005409+0000","last_peered":"2024-09-15T21:06:29.005409+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:29.005409+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43750668,"num_objects":5341,"num_object_clones":0,"num_object_copies":21364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5422","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.643212+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:31.643212+0000","last_peered":"2024-09-15T21:06:31.643212+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:31.643212+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5424","reported_seq":5488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.116037+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:33.116037+0000","last_peered":"2024-09-15T21:06:33.116037+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:33.116037+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5422,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":708011688,"num_objects":86432,"num_object_clones":0,"num_object_copies":345728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86459,"num_write_kb":691554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273827328,"data_stored":1273827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86486,"ondisk_log_size":86486,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667752,"kb_used_data":343128,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93704088,"statfs":{"total":96636764160,"available":95952986112,"internally_reserved":0,"allocated":351363072,"data_stored":351206122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674000,"kb_used_data":345728,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93697840,"statfs":{"total":96636764160,"available":95946588160,"internally_reserved":0,"allocated":354025472,"data_stored":353868522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205560,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210493440,"data_stored":210336490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631696,"kb_used_data":347392,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93740144,"statfs":{"total":96636764160,"available":95989907456,"internally_reserved":0,"allocated":355729408,"data_stored":355572458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356499456,"data_stored":356499456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211140608,"data_stored":211140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354488320,"data_stored":354488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351698944,"data_stored":351698944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:34.805 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:35.366 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:35.366 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:35.388 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-15T21:06:33.978259+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":708011688,"num_objects":86432,"num_object_clones":0,"num_object_copies":345728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86459,"num_write_kb":691554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86486,"ondisk_log_size":86486,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2373080,"kb_used_data":1241808,"kb_used_omap":14,"kb_used_meta":1131185,"kb_avail":375114280,"statfs":{"total":386547056640,"available":384117022720,"internally_reserved":0,"allocated":1271611392,"data_stored":1270983592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1158334002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,14,18,7,12,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26755072,"num_objects":3266,"num_object_clones":0,"num_object_copies":13064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":26128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004376"},"pg_stats":[{"pgid":"1.f","version":"31'5306","reported_seq":5345,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.871641+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:28.871641+0000","last_peered":"2024-09-15T21:06:28.871641+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:28.871641+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5306,"log_dups_size":0,"ondisk_log_size":5306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5306,"num_write_kb":42448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5470","reported_seq":5533,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.300577+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:33.300577+0000","last_peered":"2024-09-15T21:06:33.300577+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:33.300577+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":43740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5304","reported_seq":5378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.605541+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:31.605541+0000","last_peered":"2024-09-15T21:06:31.605541+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:31.605541+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5444","reported_seq":5508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.266202+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:33.266202+0000","last_peered":"2024-09-15T21:06:33.266202+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:33.266202+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5448","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:28.942840+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:28.942840+0000","last_peered":"2024-09-15T21:06:28.942840+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:28.942840+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5448,"log_dups_size":0,"ondisk_log_size":5448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44602636,"num_objects":5445,"num_object_clones":0,"num_object_copies":21780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5446,"num_write_kb":43565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3134}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5345","reported_seq":5385,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.049370+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:29.049370+0000","last_peered":"2024-09-15T21:06:29.049370+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:29.049370+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43786240,"num_objects":5345,"num_object_clones":0,"num_object_copies":21380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5345,"num_write_kb":42760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3029}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5581","reported_seq":5645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.243495+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:33.243495+0000","last_peered":"2024-09-15T21:06:33.243495+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:33.243495+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667596,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5569","reported_seq":5633,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.254844+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:33.254844+0000","last_peered":"2024-09-15T21:06:33.254844+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:33.254844+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577484,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5337","reported_seq":5391,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.041538+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:29.041538+0000","last_peered":"2024-09-15T21:06:29.041538+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:29.041538+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43649560,"num_objects":5329,"num_object_clones":0,"num_object_copies":21316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5334,"num_write_kb":42649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2997}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5442","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.469345+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:31.469345+0000","last_peered":"2024-09-15T21:06:31.469345+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:31.469345+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3105}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5322","reported_seq":9036,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.066248+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:29.066248+0000","last_peered":"2024-09-15T21:06:29.066248+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:29.066248+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5322,"log_dups_size":0,"ondisk_log_size":5322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43552658,"num_objects":5317,"num_object_clones":0,"num_object_copies":21268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":434,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5319,"num_write_kb":42540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1832,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":4883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5302","reported_seq":5336,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.065188+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:29.065188+0000","last_peered":"2024-09-15T21:06:29.065188+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:29.065188+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5302,"log_dups_size":0,"ondisk_log_size":5302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5301,"num_write_kb":42407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3045}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5425","reported_seq":5500,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.633046+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:31.633046+0000","last_peered":"2024-09-15T21:06:31.633046+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:31.633046+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44432006,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5345","reported_seq":5397,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:29.005409+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:29.005409+0000","last_peered":"2024-09-15T21:06:29.005409+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:29.005409+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43750668,"num_objects":5341,"num_object_clones":0,"num_object_copies":21364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5343,"num_write_kb":42740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3062}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5422","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.643212+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:31.643212+0000","last_peered":"2024-09-15T21:06:31.643212+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:31.643212+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5424","reported_seq":5488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.116037+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:33.116037+0000","last_peered":"2024-09-15T21:06:33.116037+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:33.116037+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5422,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":708011688,"num_objects":86432,"num_object_clones":0,"num_object_copies":345728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":86459,"num_write_kb":691554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":14958592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1273827328,"data_stored":1273827328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86486,"ondisk_log_size":86486,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667752,"kb_used_data":343128,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93704088,"statfs":{"total":96636764160,"available":95952986112,"internally_reserved":0,"allocated":351363072,"data_stored":351206122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,8,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674000,"kb_used_data":345728,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93697840,"statfs":{"total":96636764160,"available":95946588160,"internally_reserved":0,"allocated":354025472,"data_stored":353868522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205560,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210493440,"data_stored":210336490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631696,"kb_used_data":347392,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93740144,"statfs":{"total":96636764160,"available":95989907456,"internally_reserved":0,"allocated":355729408,"data_stored":355572458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356499456,"data_stored":356499456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211140608,"data_stored":211140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354488320,"data_stored":354488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351698944,"data_stored":351698944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5306", 'reported_seq': 5345, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:28.871641+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:28.871641+0000', 'last_peered': '2024-09-15T21:06:28.871641+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:28.871641+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5306, 'log_dups_size': 0, 'ondisk_log_size': 5306, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43466752, 'num_objects': 5306, 'num_object_clones': 0, 'num_object_copies': 21224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5306, 'num_write_kb': 42448, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3030}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5470", 'reported_seq': 5533, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:33.300577+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:33.300577+0000', 'last_peered': '2024-09-15T21:06:33.300577+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:33.300577+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5470, 'log_dups_size': 0, 'ondisk_log_size': 5470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44774668, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 21864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5468, 'num_write_kb': 43740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5304", 'reported_seq': 5378, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:31.605541+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:31.605541+0000', 'last_peered': '2024-09-15T21:06:31.605541+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:31.605541+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5304, 'log_dups_size': 0, 'ondisk_log_size': 5304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43424390, 'num_objects': 5301, 'num_object_clones': 0, 'num_object_copies': 21204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5303, 'num_write_kb': 42414, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3043}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:35.389 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5444", 'reported_seq': 5508, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:33.266202+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:33.266202+0000', 'last_peered': '2024-09-15T21:06:33.266202+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:33.266202+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5444, 'log_dups_size': 0, 'ondisk_log_size': 5444, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44561676, 'num_objects': 5440, 'num_object_clones': 0, 'num_object_copies': 21760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5442, 'num_write_kb': 43525, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3106}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5448", 'reported_seq': 5499, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:28.942840+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:28.942840+0000', 'last_peered': '2024-09-15T21:06:28.942840+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:28.942840+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5448, 'log_dups_size': 0, 'ondisk_log_size': 5448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44602636, 'num_objects': 5445, 'num_object_clones': 0, 'num_object_copies': 21780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5446, 'num_write_kb': 43565, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3134}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5345", 'reported_seq': 5385, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:29.049370+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:29.049370+0000', 'last_peered': '2024-09-15T21:06:29.049370+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:29.049370+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5345, 'log_dups_size': 0, 'ondisk_log_size': 5345, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43786240, 'num_objects': 5345, 'num_object_clones': 0, 'num_object_copies': 21380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5345, 'num_write_kb': 42760, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3029}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5581", 'reported_seq': 5645, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:33.243495+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:33.243495+0000', 'last_peered': '2024-09-15T21:06:33.243495+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:33.243495+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5581, 'log_dups_size': 0, 'ondisk_log_size': 5581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45667596, 'num_objects': 5575, 'num_object_clones': 0, 'num_object_copies': 22300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5579, 'num_write_kb': 44612, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3209}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5569", 'reported_seq': 5633, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:33.254844+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:33.254844+0000', 'last_peered': '2024-09-15T21:06:33.254844+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:33.254844+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5569, 'log_dups_size': 0, 'ondisk_log_size': 5569, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45577484, 'num_objects': 5564, 'num_object_clones': 0, 'num_object_copies': 22256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5567, 'num_write_kb': 44524, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3090}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:35.390 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5337", 'reported_seq': 5391, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:29.041538+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:29.041538+0000', 'last_peered': '2024-09-15T21:06:29.041538+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:29.041538+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5337, 'log_dups_size': 0, 'ondisk_log_size': 5337, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43649560, 'num_objects': 5329, 'num_object_clones': 0, 'num_object_copies': 21316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5334, 'num_write_kb': 42649, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2997}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5442", 'reported_seq': 5518, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:31.469345+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:31.469345+0000', 'last_peered': '2024-09-15T21:06:31.469345+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:31.469345+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5442, 'log_dups_size': 0, 'ondisk_log_size': 5442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44561676, 'num_objects': 5440, 'num_object_clones': 0, 'num_object_copies': 21760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5440, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5440, 'num_write_kb': 43518, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3105}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5322", 'reported_seq': 9036, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:29.066248+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:29.066248+0000', 'last_peered': '2024-09-15T21:06:29.066248+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:29.066248+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5322, 'log_dups_size': 0, 'ondisk_log_size': 5322, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43552658, 'num_objects': 5317, 'num_object_clones': 0, 'num_object_copies': 21268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 434, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5319, 'num_write_kb': 42540, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1832, 'num_bytes_recovered': 14958592, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 4883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5302", 'reported_seq': 5336, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:29.065188+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:29.065188+0000', 'last_peered': '2024-09-15T21:06:29.065188+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:29.065188+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5302, 'log_dups_size': 0, 'ondisk_log_size': 5302, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43424390, 'num_objects': 5301, 'num_object_clones': 0, 'num_object_copies': 21204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5301, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5301, 'num_write_kb': 42407, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3045}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5425", 'reported_seq': 5500, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:31.633046+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:31.633046+0000', 'last_peered': '2024-09-15T21:06:31.633046+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:31.633046+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5425, 'log_dups_size': 0, 'ondisk_log_size': 5425, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44432006, 'num_objects': 5424, 'num_object_clones': 0, 'num_object_copies': 21696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5424, 'num_write_kb': 43391, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:35.391 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5345", 'reported_seq': 5397, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:29.005409+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:29.005409+0000', 'last_peered': '2024-09-15T21:06:29.005409+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:29.005409+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5345, 'log_dups_size': 0, 'ondisk_log_size': 5345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43750668, 'num_objects': 5341, 'num_object_clones': 0, 'num_object_copies': 21364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5341, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5343, 'num_write_kb': 42740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3062}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5422", 'reported_seq': 5499, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:31.643212+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:31.643212+0000', 'last_peered': '2024-09-15T21:06:31.643212+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:31.643212+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5422, 'log_dups_size': 0, 'ondisk_log_size': 5422, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44389644, 'num_objects': 5419, 'num_object_clones': 0, 'num_object_copies': 21676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5420, 'num_write_kb': 43357, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:35.392 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5424", 'reported_seq': 5488, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:33.116037+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:33.116037+0000', 'last_peered': '2024-09-15T21:06:33.116037+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:33.116037+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5424, 'log_dups_size': 0, 'ondisk_log_size': 5424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44389644, 'num_objects': 5419, 'num_object_clones': 0, 'num_object_copies': 21676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5419, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5422, 'num_write_kb': 43364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:35.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:35.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 192 20 87656 87636 3.56535 2 0.0456535 0.0438084 2024-09-15T21:06:36.126 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:36.126 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35257/345728 objects degraded (10.198%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":261,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":86432,"data_bytes":708011688,"bytes_used":2430033920,"bytes_avail":384117022720,"bytes_total":386547056640,"degraded_objects":35257,"degraded_total":345728,"degraded_ratio":0.10197901240281376,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":98268,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"write_bytes_sec":2228776,"read_op_per_sec":0,"write_op_per_sec":272},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:36.155 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:36.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:36.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 193 20 87915 87895 3.55736 2.02344 0.146458 0.0439083 2024-09-15T21:06:36.778 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:36.778 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:36.801 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-15T21:06:35.979218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":712910504,"num_objects":87030,"num_object_clones":0,"num_object_copies":348120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87057,"num_write_kb":696338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":15441920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87084,"ondisk_log_size":87084,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2382712,"kb_used_data":1247280,"kb_used_omap":14,"kb_used_meta":1135345,"kb_avail":375104648,"statfs":{"total":386547056640,"available":384107159552,"internally_reserved":0,"allocated":1277214720,"data_stored":1276586920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1162593842},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,14,12,8,12,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28172288,"num_objects":3439,"num_object_clones":0,"num_object_copies":13756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-203,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3439,"num_write_kb":27512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":1662976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004532"},"pg_stats":[{"pgid":"1.f","version":"31'5387","reported_seq":5427,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:34.059617+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:34.059617+0000","last_peered":"2024-09-15T21:06:34.059617+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:34.059617+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44130304,"num_objects":5387,"num_object_clones":0,"num_object_copies":21548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5387,"num_write_kb":43096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5470","reported_seq":5533,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.300577+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:33.300577+0000","last_peered":"2024-09-15T21:06:33.300577+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:33.300577+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5468,"num_write_kb":43740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5304","reported_seq":5378,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.605541+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:31.605541+0000","last_peered":"2024-09-15T21:06:31.605541+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:31.605541+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5304,"log_dups_size":0,"ondisk_log_size":5304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43424390,"num_objects":5301,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":42414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3043}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5444","reported_seq":5508,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.266202+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:33.266202+0000","last_peered":"2024-09-15T21:06:33.266202+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:33.266202+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5444,"log_dups_size":0,"ondisk_log_size":5444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5442,"num_write_kb":43525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3106}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5543","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.964694+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:33.964694+0000","last_peered":"2024-09-15T21:06:33.964694+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:33.964694+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380876,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":44325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5424","reported_seq":5464,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.945454+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:33.945454+0000","last_peered":"2024-09-15T21:06:33.945454+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:33.945454+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433408,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3108}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5581","reported_seq":5645,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.243495+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:33.243495+0000","last_peered":"2024-09-15T21:06:33.243495+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:33.243495+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5581,"log_dups_size":0,"ondisk_log_size":5581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45667596,"num_objects":5575,"num_object_clones":0,"num_object_copies":22300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5579,"num_write_kb":44612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3209}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5569","reported_seq":5633,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.254844+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:33.254844+0000","last_peered":"2024-09-15T21:06:33.254844+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:33.254844+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45577484,"num_objects":5564,"num_object_clones":0,"num_object_copies":22256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5567,"num_write_kb":44524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3090}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5421","reported_seq":5476,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.833680+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:33.833680+0000","last_peered":"2024-09-15T21:06:33.833680+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:33.833680+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44337688,"num_objects":5413,"num_object_clones":0,"num_object_copies":21652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":43321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3081}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5442","reported_seq":5518,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.469345+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:31.469345+0000","last_peered":"2024-09-15T21:06:31.469345+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:31.469345+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44561676,"num_objects":5440,"num_object_clones":0,"num_object_copies":21760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5440,"num_write_kb":43518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3105}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5411","reported_seq":9242,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:34.068223+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:34.068223+0000","last_peered":"2024-09-15T21:06:34.068223+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:34.068223+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44281746,"num_objects":5406,"num_object_clones":0,"num_object_copies":21624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":375,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5408,"num_write_kb":43252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1891,"num_bytes_recovered":15441920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5031}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5400","reported_seq":5435,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:34.048190+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:34.048190+0000","last_peered":"2024-09-15T21:06:34.048190+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:34.048190+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5400,"log_dups_size":0,"ondisk_log_size":5400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44227206,"num_objects":5399,"num_object_clones":0,"num_object_copies":21596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5399,"num_write_kb":43191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3143}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5425","reported_seq":5500,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.633046+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:31.633046+0000","last_peered":"2024-09-15T21:06:31.633046+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:31.633046+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44432006,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5424,"num_write_kb":43391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5417","reported_seq":5469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.968219+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:33.968219+0000","last_peered":"2024-09-15T21:06:33.968219+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:33.968219+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5417,"log_dups_size":0,"ondisk_log_size":5417,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44340492,"num_objects":5413,"num_object_clones":0,"num_object_copies":21652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5415,"num_write_kb":43316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5422","reported_seq":5499,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:31.643212+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:31.643212+0000","last_peered":"2024-09-15T21:06:31.643212+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:31.643212+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5422,"log_dups_size":0,"ondisk_log_size":5422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5420,"num_write_kb":43357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3138}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5424","reported_seq":5488,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:33.116037+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:33.116037+0000","last_peered":"2024-09-15T21:06:33.116037+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:33.116037+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5424,"log_dups_size":0,"ondisk_log_size":5424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44389644,"num_objects":5419,"num_object_clones":0,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5422,"num_write_kb":43364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":712910504,"num_objects":87030,"num_object_clones":0,"num_object_copies":348120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35198,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87057,"num_write_kb":696338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":15441920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1279528960,"data_stored":1279528960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87084,"ondisk_log_size":87084,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":677384,"kb_used_data":348600,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93694456,"statfs":{"total":96636764160,"available":95943122944,"internally_reserved":0,"allocated":356966400,"data_stored":356809450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674000,"kb_used_data":345728,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93697840,"statfs":{"total":96636764160,"available":95946588160,"internally_reserved":0,"allocated":354025472,"data_stored":353868522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051616,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":399632,"kb_used_data":205560,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":93972208,"statfs":{"total":96636764160,"available":96227540992,"internally_reserved":0,"allocated":210493440,"data_stored":210336490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":198700751},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":631696,"kb_used_data":347392,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93740144,"statfs":{"total":96636764160,"available":95989907456,"internally_reserved":0,"allocated":355729408,"data_stored":355572458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":291106510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,7,6,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":356499456,"data_stored":356499456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":211140608,"data_stored":211140608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354488320,"data_stored":354488320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":357400576,"data_stored":357400576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:37.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 194 20 88197 88177 3.55038 2.20312 0.0654461 0.0439912 2024-09-15T21:06:38.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 195 20 88495 88475 3.54411 2.32812 0.0870617 0.0440763 2024-09-15T21:06:39.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 196 20 88785 88765 3.53759 2.26562 0.0588041 0.0441563 2024-09-15T21:06:39.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:40.395 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:40.396 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:40.423 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-15T21:06:39.984068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":724084392,"num_objects":88394,"num_object_clones":0,"num_object_copies":353576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88421,"num_write_kb":707250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88448,"ondisk_log_size":88448,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2406488,"kb_used_data":1269656,"kb_used_omap":14,"kb_used_meta":1136753,"kb_avail":375080872,"statfs":{"total":386547056640,"available":384082812928,"internally_reserved":0,"allocated":1300127744,"data_stored":1299499944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1164035634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,2,12,17,24,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31137792,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008134"},"pg_stats":[{"pgid":"1.f","version":"31'5475","reported_seq":5516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.992501+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:38.992501+0000","last_peered":"2024-09-15T21:06:38.992501+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:38.992501+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44851200,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5544","reported_seq":5607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.238474+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:38.238474+0000","last_peered":"2024-09-15T21:06:38.238474+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:38.238474+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380876,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5395","reported_seq":5469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.661174+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:36.661174+0000","last_peered":"2024-09-15T21:06:36.661174+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:36.661174+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44169862,"num_objects":5392,"num_object_clones":0,"num_object_copies":21568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5516","reported_seq":5580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.290722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:38.290722+0000","last_peered":"2024-09-15T21:06:38.290722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:38.290722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151500,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5514,"num_write_kb":44101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5625","reported_seq":5674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.966434+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:38.966434+0000","last_peered":"2024-09-15T21:06:38.966434+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:38.966434+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5513","reported_seq":5553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.936003+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:38.936003+0000","last_peered":"2024-09-15T21:06:38.936003+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:38.936003+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45162496,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5673","reported_seq":5738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.150652+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:38.150652+0000","last_peered":"2024-09-15T21:06:38.150652+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:38.150652+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5647","reported_seq":5710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.314021+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:38.314021+0000","last_peered":"2024-09-15T21:06:38.314021+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:38.314021+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46216460,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5503","reported_seq":5558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.019906+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:39.019906+0000","last_peered":"2024-09-15T21:06:39.019906+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:39.019906+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45009432,"num_objects":5495,"num_object_clones":0,"num_object_copies":21980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":43977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5532","reported_seq":5608,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.635596+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:36.635596+0000","last_peered":"2024-09-15T21:06:36.635596+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:36.635596+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45298956,"num_objects":5530,"num_object_clones":0,"num_object_copies":22120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5495","reported_seq":9483,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.087886+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:39.087886+0000","last_peered":"2024-09-15T21:06:39.087886+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:39.087886+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44969874,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1969,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5488","reported_seq":5521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.106102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:39.106102+0000","last_peered":"2024-09-15T21:06:39.106102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:39.106102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948102,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5487,"num_write_kb":43895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5509","reported_seq":5584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.514772+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:36.514772+0000","last_peered":"2024-09-15T21:06:36.514772+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:36.514772+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45120134,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5508,"num_write_kb":44063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5503","reported_seq":5555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.050449+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:39.050449+0000","last_peered":"2024-09-15T21:06:39.050449+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:39.050449+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45045004,"num_objects":5499,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5501,"num_write_kb":44004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5517","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.541334+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:36.541334+0000","last_peered":"2024-09-15T21:06:36.541334+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:36.541334+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5513","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.272754+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:38.272754+0000","last_peered":"2024-09-15T21:06:38.272754+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:38.272754+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118732,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5511,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":724084392,"num_objects":88394,"num_object_clones":0,"num_object_copies":353576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88421,"num_write_kb":707250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1302335488,"data_stored":1302335488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88448,"ondisk_log_size":88448,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670340,"kb_used_data":354036,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93701500,"statfs":{"total":96636764160,"available":95950336000,"internally_reserved":0,"allocated":362532864,"data_stored":362375914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667112,"kb_used_data":351320,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93704728,"statfs":{"total":96636764160,"available":95953641472,"internally_reserved":0,"allocated":359751680,"data_stored":359594730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409660,"kb_used_data":211436,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93962180,"statfs":{"total":96636764160,"available":96217272320,"internally_reserved":0,"allocated":216510464,"data_stored":216353514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659376,"kb_used_data":352864,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93712464,"statfs":{"total":96636764160,"available":95961563136,"internally_reserved":0,"allocated":361332736,"data_stored":361175786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":313847502},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362115072,"data_stored":362115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217079808,"data_stored":217079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360120320,"data_stored":360120320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363020288,"data_stored":363020288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:40.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:40.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 197 20 89051 89031 3.53018 2.07812 0.112077 0.0442484 2024-09-15T21:06:41.052 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:41.052 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:41.087 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-15T21:06:39.984068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":724084392,"num_objects":88394,"num_object_clones":0,"num_object_copies":353576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88421,"num_write_kb":707250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88448,"ondisk_log_size":88448,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2406488,"kb_used_data":1269656,"kb_used_omap":14,"kb_used_meta":1136753,"kb_avail":375080872,"statfs":{"total":386547056640,"available":384082812928,"internally_reserved":0,"allocated":1300127744,"data_stored":1299499944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1164035634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,6,2,12,17,24,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31137792,"num_objects":3801,"num_object_clones":0,"num_object_copies":15204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3801,"num_write_kb":30408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008134"},"pg_stats":[{"pgid":"1.f","version":"31'5475","reported_seq":5516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.992501+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:38.992501+0000","last_peered":"2024-09-15T21:06:38.992501+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:38.992501+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44851200,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5544","reported_seq":5607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.238474+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:38.238474+0000","last_peered":"2024-09-15T21:06:38.238474+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:38.238474+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380876,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5395","reported_seq":5469,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.661174+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:36.661174+0000","last_peered":"2024-09-15T21:06:36.661174+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:36.661174+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44169862,"num_objects":5392,"num_object_clones":0,"num_object_copies":21568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5394,"num_write_kb":43142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3134}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5516","reported_seq":5580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.290722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:38.290722+0000","last_peered":"2024-09-15T21:06:38.290722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:38.290722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151500,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5514,"num_write_kb":44101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5625","reported_seq":5674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.966434+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:38.966434+0000","last_peered":"2024-09-15T21:06:38.966434+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:38.966434+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5513","reported_seq":5553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.936003+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:38.936003+0000","last_peered":"2024-09-15T21:06:38.936003+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:38.936003+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45162496,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5673","reported_seq":5738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.150652+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:38.150652+0000","last_peered":"2024-09-15T21:06:38.150652+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:38.150652+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5647","reported_seq":5710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.314021+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:38.314021+0000","last_peered":"2024-09-15T21:06:38.314021+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:38.314021+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46216460,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5503","reported_seq":5558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.019906+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:39.019906+0000","last_peered":"2024-09-15T21:06:39.019906+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:39.019906+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45009432,"num_objects":5495,"num_object_clones":0,"num_object_copies":21980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":43977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5532","reported_seq":5608,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.635596+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:36.635596+0000","last_peered":"2024-09-15T21:06:36.635596+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:36.635596+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5532,"log_dups_size":0,"ondisk_log_size":5532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45298956,"num_objects":5530,"num_object_clones":0,"num_object_copies":22120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5530,"num_write_kb":44238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3195}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5495","reported_seq":9483,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.087886+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:39.087886+0000","last_peered":"2024-09-15T21:06:39.087886+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:39.087886+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44969874,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1969,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5488","reported_seq":5521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.106102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:39.106102+0000","last_peered":"2024-09-15T21:06:39.106102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:39.106102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948102,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5487,"num_write_kb":43895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5509","reported_seq":5584,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.514772+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:36.514772+0000","last_peered":"2024-09-15T21:06:36.514772+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:36.514772+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45120134,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5508,"num_write_kb":44063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3222}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5503","reported_seq":5555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.050449+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:39.050449+0000","last_peered":"2024-09-15T21:06:39.050449+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:39.050449+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45045004,"num_objects":5499,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5501,"num_write_kb":44004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5517","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:36.541334+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:36.541334+0000","last_peered":"2024-09-15T21:06:36.541334+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:36.541334+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5517,"log_dups_size":0,"ondisk_log_size":5517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5515,"num_write_kb":44117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5513","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.272754+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:38.272754+0000","last_peered":"2024-09-15T21:06:38.272754+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:38.272754+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118732,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5511,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":724084392,"num_objects":88394,"num_object_clones":0,"num_object_copies":353576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88421,"num_write_kb":707250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1302335488,"data_stored":1302335488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88448,"ondisk_log_size":88448,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670340,"kb_used_data":354036,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93701500,"statfs":{"total":96636764160,"available":95950336000,"internally_reserved":0,"allocated":362532864,"data_stored":362375914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":667112,"kb_used_data":351320,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93704728,"statfs":{"total":96636764160,"available":95953641472,"internally_reserved":0,"allocated":359751680,"data_stored":359594730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,4,2,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409660,"kb_used_data":211436,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93962180,"statfs":{"total":96636764160,"available":96217272320,"internally_reserved":0,"allocated":216510464,"data_stored":216353514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659376,"kb_used_data":352864,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93712464,"statfs":{"total":96636764160,"available":95961563136,"internally_reserved":0,"allocated":361332736,"data_stored":361175786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":313847502},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362115072,"data_stored":362115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217079808,"data_stored":217079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":360120320,"data_stored":360120320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363020288,"data_stored":363020288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5475", 'reported_seq': 5516, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.992501+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:38.992501+0000', 'last_peered': '2024-09-15T21:06:38.992501+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:38.992501+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5475, 'log_dups_size': 0, 'ondisk_log_size': 5475, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44851200, 'num_objects': 5475, 'num_object_clones': 0, 'num_object_copies': 21900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5475, 'num_write_kb': 43800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3199}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5544", 'reported_seq': 5607, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.238474+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:38.238474+0000', 'last_peered': '2024-09-15T21:06:38.238474+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:38.238474+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5544, 'log_dups_size': 0, 'ondisk_log_size': 5544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45380876, 'num_objects': 5540, 'num_object_clones': 0, 'num_object_copies': 22160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5542, 'num_write_kb': 44332, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3160}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5395", 'reported_seq': 5469, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:36.661174+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:36.661174+0000', 'last_peered': '2024-09-15T21:06:36.661174+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:36.661174+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5395, 'log_dups_size': 0, 'ondisk_log_size': 5395, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44169862, 'num_objects': 5392, 'num_object_clones': 0, 'num_object_copies': 21568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5392, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5394, 'num_write_kb': 43142, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3134}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5516", 'reported_seq': 5580, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.290722+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:38.290722+0000', 'last_peered': '2024-09-15T21:06:38.290722+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:38.290722+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5516, 'log_dups_size': 0, 'ondisk_log_size': 5516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45151500, 'num_objects': 5512, 'num_object_clones': 0, 'num_object_copies': 22048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5514, 'num_write_kb': 44101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5625", 'reported_seq': 5674, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.966434+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:38.966434+0000', 'last_peered': '2024-09-15T21:06:38.966434+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:38.966434+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5625, 'log_dups_size': 0, 'ondisk_log_size': 5625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46052620, 'num_objects': 5622, 'num_object_clones': 0, 'num_object_copies': 22488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5623, 'num_write_kb': 44981, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3311}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5513", 'reported_seq': 5553, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.936003+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:38.936003+0000', 'last_peered': '2024-09-15T21:06:38.936003+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:38.936003+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5513, 'log_dups_size': 0, 'ondisk_log_size': 5513, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45162496, 'num_objects': 5513, 'num_object_clones': 0, 'num_object_copies': 22052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5513, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5513, 'num_write_kb': 44104, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3197}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5673", 'reported_seq': 5738, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.150652+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:38.150652+0000', 'last_peered': '2024-09-15T21:06:38.150652+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:38.150652+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5673, 'log_dups_size': 0, 'ondisk_log_size': 5673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46421260, 'num_objects': 5667, 'num_object_clones': 0, 'num_object_copies': 22668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5671, 'num_write_kb': 45348, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3301}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5647", 'reported_seq': 5710, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.314021+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:38.314021+0000', 'last_peered': '2024-09-15T21:06:38.314021+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:38.314021+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5647, 'log_dups_size': 0, 'ondisk_log_size': 5647, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46216460, 'num_objects': 5642, 'num_object_clones': 0, 'num_object_copies': 22568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5642, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5645, 'num_write_kb': 45148, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5503", 'reported_seq': 5558, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:39.019906+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:39.019906+0000', 'last_peered': '2024-09-15T21:06:39.019906+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:39.019906+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5503, 'log_dups_size': 0, 'ondisk_log_size': 5503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45009432, 'num_objects': 5495, 'num_object_clones': 0, 'num_object_copies': 21980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5495, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5500, 'num_write_kb': 43977, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3163}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5532", 'reported_seq': 5608, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:36.635596+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:36.635596+0000', 'last_peered': '2024-09-15T21:06:36.635596+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:36.635596+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5532, 'log_dups_size': 0, 'ondisk_log_size': 5532, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45298956, 'num_objects': 5530, 'num_object_clones': 0, 'num_object_copies': 22120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5530, 'num_write_kb': 44238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3195}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5495", 'reported_seq': 9483, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:39.087886+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:39.087886+0000', 'last_peered': '2024-09-15T21:06:39.087886+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:39.087886+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5495, 'log_dups_size': 0, 'ondisk_log_size': 5495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44969874, 'num_objects': 5490, 'num_object_clones': 0, 'num_object_copies': 21960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 297, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5492, 'num_write_kb': 43924, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1969, 'num_bytes_recovered': 16080896, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5193}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5488", 'reported_seq': 5521, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:39.106102+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:39.106102+0000', 'last_peered': '2024-09-15T21:06:39.106102+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:39.106102+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5488, 'log_dups_size': 0, 'ondisk_log_size': 5488, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44948102, 'num_objects': 5487, 'num_object_clones': 0, 'num_object_copies': 21948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5487, 'num_write_kb': 43895, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5509", 'reported_seq': 5584, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:36.514772+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:36.514772+0000', 'last_peered': '2024-09-15T21:06:36.514772+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:36.514772+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5509, 'log_dups_size': 0, 'ondisk_log_size': 5509, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45120134, 'num_objects': 5508, 'num_object_clones': 0, 'num_object_copies': 22032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5508, 'num_write_kb': 44063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3222}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5503", 'reported_seq': 5555, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:39.050449+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:39.050449+0000', 'last_peered': '2024-09-15T21:06:39.050449+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:39.050449+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5503, 'log_dups_size': 0, 'ondisk_log_size': 5503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45045004, 'num_objects': 5499, 'num_object_clones': 0, 'num_object_copies': 21996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5501, 'num_write_kb': 44004, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:41.090 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:41.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5517", 'reported_seq': 5594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:36.541334+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:36.541334+0000', 'last_peered': '2024-09-15T21:06:36.541334+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:36.541334+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5517, 'log_dups_size': 0, 'ondisk_log_size': 5517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45167884, 'num_objects': 5514, 'num_object_clones': 0, 'num_object_copies': 22056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5515, 'num_write_kb': 44117, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3233}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:41.091 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:41.091 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5513", 'reported_seq': 5577, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:38.272754+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:38.272754+0000', 'last_peered': '2024-09-15T21:06:38.272754+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:38.272754+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5513, 'log_dups_size': 0, 'ondisk_log_size': 5513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45118732, 'num_objects': 5508, 'num_object_clones': 0, 'num_object_copies': 22032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5508, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5511, 'num_write_kb': 44076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3193}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:41.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:41.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 198 20 89299 89279 3.52213 1.9375 0.0644322 0.0443444 2024-09-15T21:06:41.839 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:41.839 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35120/353576 objects degraded (9.933%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":267,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88394,"data_bytes":724084392,"bytes_used":2464243712,"bytes_avail":384082812928,"bytes_total":386547056640,"degraded_objects":35120,"degraded_total":353576,"degraded_ratio":0.099328008688372507,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":141216,"recovering_keys_per_sec":0,"num_objects_recovered":207,"num_bytes_recovered":1695744,"num_keys_recovered":0,"write_bytes_sec":2593058,"read_op_per_sec":0,"write_op_per_sec":316},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:41.871 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:41.872 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:42.470 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:42.472 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:42.501 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-15T21:06:41.985004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":727131816,"num_objects":88766,"num_object_clones":0,"num_object_copies":355064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88793,"num_write_kb":710226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88820,"ondisk_log_size":88820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2416160,"kb_used_data":1275168,"kb_used_omap":14,"kb_used_meta":1140913,"kb_avail":375071200,"statfs":{"total":386547056640,"available":384072908800,"internally_reserved":0,"allocated":1305772032,"data_stored":1305144232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1168295474},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,2,9,15,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25468928,"num_objects":3109,"num_object_clones":0,"num_object_copies":12436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":24872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":1122304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008374"},"pg_stats":[{"pgid":"1.f","version":"31'5475","reported_seq":5516,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.992501+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:38.992501+0000","last_peered":"2024-09-15T21:06:38.992501+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:38.992501+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44851200,"num_objects":5475,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5475,"num_write_kb":43800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5544","reported_seq":5607,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.238474+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:38.238474+0000","last_peered":"2024-09-15T21:06:38.238474+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:38.238474+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5544,"log_dups_size":0,"ondisk_log_size":5544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45380876,"num_objects":5540,"num_object_clones":0,"num_object_copies":22160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5542,"num_write_kb":44332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5489","reported_seq":5563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.548984+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:41.548984+0000","last_peered":"2024-09-15T21:06:41.548984+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:41.548984+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44939910,"num_objects":5486,"num_object_clones":0,"num_object_copies":21944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":43894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5516","reported_seq":5580,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.290722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:38.290722+0000","last_peered":"2024-09-15T21:06:38.290722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:38.290722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5516,"log_dups_size":0,"ondisk_log_size":5516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45151500,"num_objects":5512,"num_object_clones":0,"num_object_copies":22048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5514,"num_write_kb":44101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3178}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5625","reported_seq":5674,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.966434+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:38.966434+0000","last_peered":"2024-09-15T21:06:38.966434+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:38.966434+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5513","reported_seq":5553,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.936003+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:38.936003+0000","last_peered":"2024-09-15T21:06:38.936003+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:38.936003+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45162496,"num_objects":5513,"num_object_clones":0,"num_object_copies":22052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5513,"num_write_kb":44104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5673","reported_seq":5738,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.150652+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:38.150652+0000","last_peered":"2024-09-15T21:06:38.150652+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:38.150652+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5673,"log_dups_size":0,"ondisk_log_size":5673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46421260,"num_objects":5667,"num_object_clones":0,"num_object_copies":22668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":45348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3301}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5647","reported_seq":5710,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.314021+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:38.314021+0000","last_peered":"2024-09-15T21:06:38.314021+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:38.314021+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46216460,"num_objects":5642,"num_object_clones":0,"num_object_copies":22568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5645,"num_write_kb":45148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3168}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5503","reported_seq":5558,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.019906+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:39.019906+0000","last_peered":"2024-09-15T21:06:39.019906+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:39.019906+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45009432,"num_objects":5495,"num_object_clones":0,"num_object_copies":21980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5500,"num_write_kb":43977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3163}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5624","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.656231+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:41.656231+0000","last_peered":"2024-09-15T21:06:41.656231+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:41.656231+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":44974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5495","reported_seq":9483,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.087886+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:39.087886+0000","last_peered":"2024-09-15T21:06:39.087886+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:39.087886+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5495,"log_dups_size":0,"ondisk_log_size":5495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44969874,"num_objects":5490,"num_object_clones":0,"num_object_copies":21960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":297,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5492,"num_write_kb":43924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1969,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5193}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5488","reported_seq":5521,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.106102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:39.106102+0000","last_peered":"2024-09-15T21:06:39.106102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:39.106102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948102,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5487,"num_write_kb":43895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3231}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5606","reported_seq":5681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.674921+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:41.674921+0000","last_peered":"2024-09-15T21:06:41.674921+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:41.674921+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914758,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5503","reported_seq":5555,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:39.050449+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:39.050449+0000","last_peered":"2024-09-15T21:06:39.050449+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:39.050449+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45045004,"num_objects":5499,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5501,"num_write_kb":44004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5606","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.612679+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:41.612679+0000","last_peered":"2024-09-15T21:06:41.612679+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:41.612679+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896972,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5513","reported_seq":5577,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:38.272754+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:38.272754+0000","last_peered":"2024-09-15T21:06:38.272754+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:38.272754+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5513,"log_dups_size":0,"ondisk_log_size":5513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45118732,"num_objects":5508,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5511,"num_write_kb":44076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3193}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":727131816,"num_objects":88766,"num_object_clones":0,"num_object_copies":355064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88793,"num_write_kb":710226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1993,"num_bytes_recovered":16080896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1308045312,"data_stored":1308045312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88820,"ondisk_log_size":88820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":670340,"kb_used_data":354036,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93701500,"statfs":{"total":96636764160,"available":95950336000,"internally_reserved":0,"allocated":362532864,"data_stored":362375914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676784,"kb_used_data":356832,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93695056,"statfs":{"total":96636764160,"available":95943737344,"internally_reserved":0,"allocated":365395968,"data_stored":365239018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051617,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":409660,"kb_used_data":211436,"kb_used_omap":4,"kb_used_meta":198203,"kb_avail":93962180,"statfs":{"total":96636764160,"available":96217272320,"internally_reserved":0,"allocated":216510464,"data_stored":216353514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":202960591},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,5,11,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":659376,"kb_used_data":352864,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93712464,"statfs":{"total":96636764160,"available":95961563136,"internally_reserved":0,"allocated":361332736,"data_stored":361175786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":313847502},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,2,7,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362115072,"data_stored":362115072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217079808,"data_stored":217079808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365830144,"data_stored":365830144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":363020288,"data_stored":363020288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:42.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 199 20 89554 89534 3.51444 1.99219 0.039315 0.044449 2024-09-15T21:06:43.657 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:06:43.655808+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0445243 2024-09-15T21:06:43.658 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:06:43.658 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 200 20 89842 89822 3.50812 2.25 0.0953206 0.0445243 2024-09-15T21:06:44.658 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 201 20 90138 90118 3.50217 2.3125 0.0387028 0.0446057 2024-09-15T21:06:45.503 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:45.659 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 202 20 90411 90391 3.49537 2.13281 0.0586883 0.0446874 2024-09-15T21:06:46.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:46.068 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:46.104 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-15T21:06:45.986766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735250088,"num_objects":89757,"num_object_clones":0,"num_object_copies":359028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89784,"num_write_kb":718154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89811,"ondisk_log_size":89811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2481428,"kb_used_data":1291860,"kb_used_omap":14,"kb_used_meta":1189489,"kb_avail":375005932,"statfs":{"total":386547056640,"available":384006074368,"internally_reserved":0,"allocated":1322864640,"data_stored":1322236840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1218037298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,6,6,19,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27238400,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1687552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008506"},"pg_stats":[{"pgid":"1.f","version":"31'5550","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.719364+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:43.719364+0000","last_peered":"2024-09-15T21:06:43.719364+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:43.719364+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45465600,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5550,"num_write_kb":44400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5625","reported_seq":5688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.265472+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:43.265472+0000","last_peered":"2024-09-15T21:06:43.265472+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:43.265472+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46044428,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5489","reported_seq":5563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.548984+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:41.548984+0000","last_peered":"2024-09-15T21:06:41.548984+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:41.548984+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44939910,"num_objects":5486,"num_object_clones":0,"num_object_copies":21944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":43894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.345323+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:43.345323+0000","last_peered":"2024-09-15T21:06:43.345323+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:43.345323+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905164,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5708","reported_seq":5759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.085532+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:44.085532+0000","last_peered":"2024-09-15T21:06:44.085532+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:44.085532+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732556,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5578","reported_seq":5618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.056230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:44.056230+0000","last_peered":"2024-09-15T21:06:44.056230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:44.056230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5756","reported_seq":5821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.198142+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:43.198142+0000","last_peered":"2024-09-15T21:06:43.198142+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:43.198142+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101196,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3384}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5721","reported_seq":5785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.229400+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:43.229400+0000","last_peered":"2024-09-15T21:06:43.229400+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:43.229400+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822668,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5599","reported_seq":5654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.036937+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:44.036937+0000","last_peered":"2024-09-15T21:06:44.036937+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:44.036937+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45795864,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5624","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.656231+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:41.656231+0000","last_peered":"2024-09-15T21:06:41.656231+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:41.656231+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":44974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5587","reported_seq":9713,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.094682+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:44.094682+0000","last_peered":"2024-09-15T21:06:44.094682+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:44.094682+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45723538,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5559","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.921855+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:43.921855+0000","last_peered":"2024-09-15T21:06:43.921855+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:43.921855+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45529734,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5606","reported_seq":5681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.674921+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:41.674921+0000","last_peered":"2024-09-15T21:06:41.674921+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:41.674921+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914758,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5597","reported_seq":5649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.075565+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:44.075565+0000","last_peered":"2024-09-15T21:06:44.075565+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:44.075565+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5606","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.612679+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:41.612679+0000","last_peered":"2024-09-15T21:06:41.612679+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:41.612679+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896972,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5598","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.375855+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:43.375855+0000","last_peered":"2024-09-15T21:06:43.375855+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:43.375855+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735250088,"num_objects":89757,"num_object_clones":0,"num_object_copies":359028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89784,"num_write_kb":718154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1325084672,"data_stored":1325084672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89811,"ondisk_log_size":89811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679920,"kb_used_data":359456,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93691920,"statfs":{"total":96636764160,"available":95940526080,"internally_reserved":0,"allocated":368082944,"data_stored":367925994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":328134352},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,2,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676784,"kb_used_data":356832,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93695056,"statfs":{"total":96636764160,"available":95943737344,"internally_reserved":0,"allocated":365395968,"data_stored":365239018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419660,"kb_used_data":217276,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93952180,"statfs":{"total":96636764160,"available":96207032320,"internally_reserved":0,"allocated":222490624,"data_stored":222333674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705064,"kb_used_data":358296,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666776,"statfs":{"total":96636764160,"available":95914778624,"internally_reserved":0,"allocated":366895104,"data_stored":366738154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367730688,"data_stored":367730688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222973952,"data_stored":222973952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365830144,"data_stored":365830144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368549888,"data_stored":368549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:46.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:46.659 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 203 20 90662 90642 3.48781 1.96094 0.0612279 0.0447824 2024-09-15T21:06:46.683 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:46.683 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:46.712 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-15T21:06:45.986766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":735250088,"num_objects":89757,"num_object_clones":0,"num_object_copies":359028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89784,"num_write_kb":718154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89811,"ondisk_log_size":89811,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2481428,"kb_used_data":1291860,"kb_used_omap":14,"kb_used_meta":1189489,"kb_avail":375005932,"statfs":{"total":386547056640,"available":384006074368,"internally_reserved":0,"allocated":1322864640,"data_stored":1322236840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1218037298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,6,6,19,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27238400,"num_objects":3325,"num_object_clones":0,"num_object_copies":13300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-206,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3325,"num_write_kb":26600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":1687552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008506"},"pg_stats":[{"pgid":"1.f","version":"31'5550","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.719364+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:43.719364+0000","last_peered":"2024-09-15T21:06:43.719364+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:43.719364+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45465600,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5550,"num_write_kb":44400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5625","reported_seq":5688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.265472+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:43.265472+0000","last_peered":"2024-09-15T21:06:43.265472+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:43.265472+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46044428,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5489","reported_seq":5563,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.548984+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:41.548984+0000","last_peered":"2024-09-15T21:06:41.548984+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:41.548984+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5489,"log_dups_size":0,"ondisk_log_size":5489,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44939910,"num_objects":5486,"num_object_clones":0,"num_object_copies":21944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":43894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3228}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.345323+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:43.345323+0000","last_peered":"2024-09-15T21:06:43.345323+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:43.345323+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905164,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5708","reported_seq":5759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.085532+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:44.085532+0000","last_peered":"2024-09-15T21:06:44.085532+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:44.085532+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732556,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5578","reported_seq":5618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.056230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:44.056230+0000","last_peered":"2024-09-15T21:06:44.056230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:44.056230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5756","reported_seq":5821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.198142+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:43.198142+0000","last_peered":"2024-09-15T21:06:43.198142+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:43.198142+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101196,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3384}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5721","reported_seq":5785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.229400+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:43.229400+0000","last_peered":"2024-09-15T21:06:43.229400+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:43.229400+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822668,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5599","reported_seq":5654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.036937+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:44.036937+0000","last_peered":"2024-09-15T21:06:44.036937+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:44.036937+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45795864,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5624","reported_seq":5700,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.656231+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:41.656231+0000","last_peered":"2024-09-15T21:06:41.656231+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:41.656231+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46052620,"num_objects":5622,"num_object_clones":0,"num_object_copies":22488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5622,"num_write_kb":44974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3287}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5587","reported_seq":9713,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.094682+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:44.094682+0000","last_peered":"2024-09-15T21:06:44.094682+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:44.094682+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45723538,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5559","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.921855+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:43.921855+0000","last_peered":"2024-09-15T21:06:43.921855+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:43.921855+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45529734,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5606","reported_seq":5681,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.674921+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:41.674921+0000","last_peered":"2024-09-15T21:06:41.674921+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:41.674921+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45914758,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5605,"num_write_kb":44839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5597","reported_seq":5649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.075565+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:44.075565+0000","last_peered":"2024-09-15T21:06:44.075565+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:44.075565+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5606","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:41.612679+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:41.612679+0000","last_peered":"2024-09-15T21:06:41.612679+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:41.612679+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45896972,"num_objects":5603,"num_object_clones":0,"num_object_copies":22412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":44829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5598","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.375855+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:43.375855+0000","last_peered":"2024-09-15T21:06:43.375855+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:43.375855+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":735250088,"num_objects":89757,"num_object_clones":0,"num_object_copies":359028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":89784,"num_write_kb":718154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1325084672,"data_stored":1325084672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89811,"ondisk_log_size":89811,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679920,"kb_used_data":359456,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93691920,"statfs":{"total":96636764160,"available":95940526080,"internally_reserved":0,"allocated":368082944,"data_stored":367925994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":328134352},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,2,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":676784,"kb_used_data":356832,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93695056,"statfs":{"total":96636764160,"available":95943737344,"internally_reserved":0,"allocated":365395968,"data_stored":365239018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051618,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":419660,"kb_used_data":217276,"kb_used_omap":4,"kb_used_meta":202363,"kb_avail":93952180,"statfs":{"total":96636764160,"available":96207032320,"internally_reserved":0,"allocated":222490624,"data_stored":222333674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":207220431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,7,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705064,"kb_used_data":358296,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666776,"statfs":{"total":96636764160,"available":95914778624,"internally_reserved":0,"allocated":366895104,"data_stored":366738154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367730688,"data_stored":367730688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":222973952,"data_stored":222973952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":365830144,"data_stored":365830144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368549888,"data_stored":368549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:46.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:46.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5550", 'reported_seq': 5591, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.719364+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:43.719364+0000', 'last_peered': '2024-09-15T21:06:43.719364+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:43.719364+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5550, 'log_dups_size': 0, 'ondisk_log_size': 5550, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45465600, 'num_objects': 5550, 'num_object_clones': 0, 'num_object_copies': 22200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5550, 'num_write_kb': 44400, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3274}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5625", 'reported_seq': 5688, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.265472+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:43.265472+0000', 'last_peered': '2024-09-15T21:06:43.265472+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:43.265472+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5625, 'log_dups_size': 0, 'ondisk_log_size': 5625, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46044428, 'num_objects': 5621, 'num_object_clones': 0, 'num_object_copies': 22484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5621, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5623, 'num_write_kb': 44980, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5489", 'reported_seq': 5563, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:41.548984+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:41.548984+0000', 'last_peered': '2024-09-15T21:06:41.548984+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:41.548984+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5489, 'log_dups_size': 0, 'ondisk_log_size': 5489, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44939910, 'num_objects': 5486, 'num_object_clones': 0, 'num_object_copies': 21944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5488, 'num_write_kb': 43894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3228}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5608", 'reported_seq': 5670, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.345323+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:43.345323+0000', 'last_peered': '2024-09-15T21:06:43.345323+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:43.345323+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5608, 'log_dups_size': 0, 'ondisk_log_size': 5608, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45905164, 'num_objects': 5604, 'num_object_clones': 0, 'num_object_copies': 22416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5606, 'num_write_kb': 44837, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3270}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5708", 'reported_seq': 5759, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:44.085532+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:44.085532+0000', 'last_peered': '2024-09-15T21:06:44.085532+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:44.085532+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5708, 'log_dups_size': 0, 'ondisk_log_size': 5708, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46732556, 'num_objects': 5705, 'num_object_clones': 0, 'num_object_copies': 22820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5706, 'num_write_kb': 45645, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3394}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5578", 'reported_seq': 5618, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:44.056230+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:44.056230+0000', 'last_peered': '2024-09-15T21:06:44.056230+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:44.056230+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5578, 'log_dups_size': 0, 'ondisk_log_size': 5578, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45694976, 'num_objects': 5578, 'num_object_clones': 0, 'num_object_copies': 22312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5578, 'num_write_kb': 44624, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:46.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5756", 'reported_seq': 5821, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.198142+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:43.198142+0000', 'last_peered': '2024-09-15T21:06:43.198142+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:43.198142+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5756, 'log_dups_size': 0, 'ondisk_log_size': 5756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47101196, 'num_objects': 5750, 'num_object_clones': 0, 'num_object_copies': 23000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5754, 'num_write_kb': 46012, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3384}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5721", 'reported_seq': 5785, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.229400+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:43.229400+0000', 'last_peered': '2024-09-15T21:06:43.229400+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:43.229400+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5721, 'log_dups_size': 0, 'ondisk_log_size': 5721, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46822668, 'num_objects': 5716, 'num_object_clones': 0, 'num_object_copies': 22864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5719, 'num_write_kb': 45740, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5599", 'reported_seq': 5654, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:44.036937+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:44.036937+0000', 'last_peered': '2024-09-15T21:06:44.036937+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:44.036937+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5599, 'log_dups_size': 0, 'ondisk_log_size': 5599, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45795864, 'num_objects': 5591, 'num_object_clones': 0, 'num_object_copies': 22364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5591, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5596, 'num_write_kb': 44745, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5624", 'reported_seq': 5700, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:41.656231+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:41.656231+0000', 'last_peered': '2024-09-15T21:06:41.656231+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:41.656231+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5624, 'log_dups_size': 0, 'ondisk_log_size': 5624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46052620, 'num_objects': 5622, 'num_object_clones': 0, 'num_object_copies': 22488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5622, 'num_write_kb': 44974, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3287}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5587", 'reported_seq': 9713, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:44.094682+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:44.094682+0000', 'last_peered': '2024-09-15T21:06:44.094682+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:44.094682+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5587, 'log_dups_size': 0, 'ondisk_log_size': 5587, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45723538, 'num_objects': 5582, 'num_object_clones': 0, 'num_object_copies': 22328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5584, 'num_write_kb': 44660, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2038, 'num_bytes_recovered': 16646144, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5354}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.715 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5559", 'reported_seq': 5594, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.921855+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:43.921855+0000', 'last_peered': '2024-09-15T21:06:43.921855+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:43.921855+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5559, 'log_dups_size': 0, 'ondisk_log_size': 5559, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45529734, 'num_objects': 5558, 'num_object_clones': 0, 'num_object_copies': 22232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5558, 'num_write_kb': 44463, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3302}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5606", 'reported_seq': 5681, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:41.674921+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:41.674921+0000', 'last_peered': '2024-09-15T21:06:41.674921+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:41.674921+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5606, 'log_dups_size': 0, 'ondisk_log_size': 5606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45914758, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5605, 'num_write_kb': 44839, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3319}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5597", 'reported_seq': 5649, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:44.075565+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:44.075565+0000', 'last_peered': '2024-09-15T21:06:44.075565+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:44.075565+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5597, 'log_dups_size': 0, 'ondisk_log_size': 5597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45815052, 'num_objects': 5593, 'num_object_clones': 0, 'num_object_copies': 22372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5595, 'num_write_kb': 44756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3314}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:46.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:46.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5606", 'reported_seq': 5680, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:41.612679+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:41.612679+0000', 'last_peered': '2024-09-15T21:06:41.612679+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:41.612679+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5606, 'log_dups_size': 0, 'ondisk_log_size': 5606, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45896972, 'num_objects': 5603, 'num_object_clones': 0, 'num_object_copies': 22412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5603, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5604, 'num_write_kb': 44829, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:46.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:46.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5598", 'reported_seq': 5662, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:43.375855+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:43.375855+0000', 'last_peered': '2024-09-15T21:06:43.375855+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:43.375855+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5598, 'log_dups_size': 0, 'ondisk_log_size': 5598, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45815052, 'num_objects': 5593, 'num_object_clones': 0, 'num_object_copies': 22372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5596, 'num_write_kb': 44756, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3278}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:46.717 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:47.479 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:47.479 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 35051/359028 objects degraded (9.763%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89757,"data_bytes":735250088,"bytes_used":2540982272,"bytes_avail":384006074368,"bytes_total":386547056640,"degraded_objects":35051,"degraded_total":359028,"degraded_ratio":0.097627483093240638,"recovering_objects_per_sec":17,"recovering_bytes_per_sec":140529,"recovering_keys_per_sec":0,"num_objects_recovered":206,"num_bytes_recovered":1687552,"num_keys_recovered":0,"write_bytes_sec":2268258,"read_op_per_sec":0,"write_op_per_sec":276},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:47.518 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:47.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:47.659 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 204 20 90915 90895 3.4804 1.97656 0.0672933 0.044879 2024-09-15T21:06:48.107 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:48.107 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:48.133 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-15T21:06:47.987449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":738035368,"num_objects":90097,"num_object_clones":0,"num_object_copies":360388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90124,"num_write_kb":720874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90151,"ondisk_log_size":90151,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2518932,"kb_used_data":1302996,"kb_used_omap":14,"kb_used_meta":1215857,"kb_avail":374968428,"statfs":{"total":386547056640,"available":383967670272,"internally_reserved":0,"allocated":1334267904,"data_stored":1333640104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1245038130},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,8,8,19,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25124864,"num_objects":3067,"num_object_clones":0,"num_object_copies":12268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":24536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1204224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008230"},"pg_stats":[{"pgid":"1.f","version":"31'5550","reported_seq":5591,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.719364+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:43.719364+0000","last_peered":"2024-09-15T21:06:43.719364+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:43.719364+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5550,"log_dups_size":0,"ondisk_log_size":5550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45465600,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5550,"num_write_kb":44400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3274}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5625","reported_seq":5688,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.265472+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:43.265472+0000","last_peered":"2024-09-15T21:06:43.265472+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:43.265472+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46044428,"num_objects":5621,"num_object_clones":0,"num_object_copies":22484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":44980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5582","reported_seq":5656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.616188+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:46.616188+0000","last_peered":"2024-09-15T21:06:46.616188+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:46.616188+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45701766,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5608","reported_seq":5670,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.345323+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:43.345323+0000","last_peered":"2024-09-15T21:06:43.345323+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:43.345323+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5608,"log_dups_size":0,"ondisk_log_size":5608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45905164,"num_objects":5604,"num_object_clones":0,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5606,"num_write_kb":44837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5708","reported_seq":5759,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.085532+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:44.085532+0000","last_peered":"2024-09-15T21:06:44.085532+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:44.085532+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46732556,"num_objects":5705,"num_object_clones":0,"num_object_copies":22820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3394}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5578","reported_seq":5618,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.056230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:44.056230+0000","last_peered":"2024-09-15T21:06:44.056230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:44.056230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45694976,"num_objects":5578,"num_object_clones":0,"num_object_copies":22312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5578,"num_write_kb":44624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5756","reported_seq":5821,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.198142+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:43.198142+0000","last_peered":"2024-09-15T21:06:43.198142+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:43.198142+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5756,"log_dups_size":0,"ondisk_log_size":5756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47101196,"num_objects":5750,"num_object_clones":0,"num_object_copies":23000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3384}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5721","reported_seq":5785,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.229400+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:43.229400+0000","last_peered":"2024-09-15T21:06:43.229400+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:43.229400+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46822668,"num_objects":5716,"num_object_clones":0,"num_object_copies":22864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":45740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5599","reported_seq":5654,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.036937+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:44.036937+0000","last_peered":"2024-09-15T21:06:44.036937+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:44.036937+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45795864,"num_objects":5591,"num_object_clones":0,"num_object_copies":22364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5708","reported_seq":5783,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.743669+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:46.743669+0000","last_peered":"2024-09-15T21:06:46.743669+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:46.743669+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46740748,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5587","reported_seq":9713,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.094682+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:44.094682+0000","last_peered":"2024-09-15T21:06:44.094682+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:44.094682+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45723538,"num_objects":5582,"num_object_clones":0,"num_object_copies":22328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5584,"num_write_kb":44660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5354}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5559","reported_seq":5594,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.921855+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:43.921855+0000","last_peered":"2024-09-15T21:06:43.921855+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:43.921855+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45529734,"num_objects":5558,"num_object_clones":0,"num_object_copies":22232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5558,"num_write_kb":44463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3302}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5680","reported_seq":5755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.699088+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:46.699088+0000","last_peered":"2024-09-15T21:06:46.699088+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:46.699088+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46520966,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5679,"num_write_kb":45431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5597","reported_seq":5649,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:44.075565+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:44.075565+0000","last_peered":"2024-09-15T21:06:44.075565+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:44.075565+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3314}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5695","reported_seq":5773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.605183+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:46.605183+0000","last_peered":"2024-09-15T21:06:46.605183+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:46.605183+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46626060,"num_objects":5692,"num_object_clones":0,"num_object_copies":22768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5693,"num_write_kb":45541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5598","reported_seq":5662,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:43.375855+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:43.375855+0000","last_peered":"2024-09-15T21:06:43.375855+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:43.375855+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45815052,"num_objects":5593,"num_object_clones":0,"num_object_copies":22372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5596,"num_write_kb":44756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3278}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":738035368,"num_objects":90097,"num_object_clones":0,"num_object_copies":360388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35051,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90124,"num_write_kb":720874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2062,"num_bytes_recovered":16646144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1336590336,"data_stored":1336590336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90151,"ondisk_log_size":90151,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679920,"kb_used_data":359456,"kb_used_omap":4,"kb_used_meta":320443,"kb_avail":93691920,"statfs":{"total":96636764160,"available":95940526080,"internally_reserved":0,"allocated":368082944,"data_stored":367925994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":328134352},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,5,2,10,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686408,"kb_used_data":362296,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93685432,"statfs":{"total":96636764160,"available":95933882368,"internally_reserved":0,"allocated":370991104,"data_stored":370834154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447540,"kb_used_data":222948,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93924300,"statfs":{"total":96636764160,"available":96178483200,"internally_reserved":0,"allocated":228298752,"data_stored":228141802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705064,"kb_used_data":358296,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93666776,"statfs":{"total":96636764160,"available":95914778624,"internally_reserved":0,"allocated":366895104,"data_stored":366738154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367730688,"data_stored":367730688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228876288,"data_stored":228876288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371433472,"data_stored":371433472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":368549888,"data_stored":368549888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:48.660 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 205 20 91177 91157 3.47341 2.04688 0.067255 0.0449697 2024-09-15T21:06:49.659 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 206 20 91462 91442 3.46735 2.22656 0.050564 0.0450451 2024-09-15T21:06:50.660 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 207 20 91747 91727 3.46136 2.22656 0.0614019 0.0451307 2024-09-15T21:06:51.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:51.660 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 208 20 92014 91994 3.45475 2.08594 0.0735013 0.0452164 2024-09-15T21:06:51.752 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:51.753 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:51.776 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-09-15T21:06:49.988167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":746309288,"num_objects":91107,"num_object_clones":0,"num_object_copies":364428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91134,"num_write_kb":728954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91161,"ondisk_log_size":91161,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2521492,"kb_used_data":1313872,"kb_used_omap":14,"kb_used_meta":1207537,"kb_avail":374965868,"statfs":{"total":386547056640,"available":383965048832,"internally_reserved":0,"allocated":1345404928,"data_stored":1344777128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1236518450},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,12,19,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":30449664,"num_objects":3717,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":29736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":1736704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005053"},"pg_stats":[{"pgid":"1.f","version":"31'5639","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.164971+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:49.164971+0000","last_peered":"2024-09-15T21:06:49.164971+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:49.164971+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5714","reported_seq":5775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.226528+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:48.226528+0000","last_peered":"2024-09-15T21:06:48.226528+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:48.226528+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773516,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5712,"num_write_kb":45692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5582","reported_seq":5656,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.616188+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:46.616188+0000","last_peered":"2024-09-15T21:06:46.616188+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:46.616188+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5582,"log_dups_size":0,"ondisk_log_size":5582,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45701766,"num_objects":5579,"num_object_clones":0,"num_object_copies":22316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5581,"num_write_kb":44638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3321}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5677","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.360185+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:48.360185+0000","last_peered":"2024-09-15T21:06:48.360185+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:48.360185+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470412,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5803","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.935433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:48.935433+0000","last_peered":"2024-09-15T21:06:48.935433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:48.935433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47510796,"num_objects":5800,"num_object_clones":0,"num_object_copies":23200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5658","reported_seq":5697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.159053+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:49.159053+0000","last_peered":"2024-09-15T21:06:49.159053+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:49.159053+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46350336,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5832","reported_seq":5897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.355753+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:48.355753+0000","last_peered":"2024-09-15T21:06:48.355753+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:48.355753+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3460}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5810","reported_seq":5874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.375265+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:48.375265+0000","last_peered":"2024-09-15T21:06:48.375265+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:48.375265+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47551756,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5808,"num_write_kb":46452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5678","reported_seq":5733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.035206+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:49.035206+0000","last_peered":"2024-09-15T21:06:49.035206+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:49.035206+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46443032,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5708","reported_seq":5783,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.743669+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:46.743669+0000","last_peered":"2024-09-15T21:06:46.743669+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:46.743669+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5708,"log_dups_size":0,"ondisk_log_size":5708,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46740748,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5706,"num_write_kb":45646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5665","reported_seq":9920,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.158300+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:49.158300+0000","last_peered":"2024-09-15T21:06:49.158300+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:49.158300+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46362514,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2103,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5660","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.085632+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:49.085632+0000","last_peered":"2024-09-15T21:06:49.085632+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:49.085632+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46357126,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5680","reported_seq":5755,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.699088+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:46.699088+0000","last_peered":"2024-09-15T21:06:46.699088+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:46.699088+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5680,"log_dups_size":0,"ondisk_log_size":5680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46520966,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5679,"num_write_kb":45431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5676","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.988065+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:48.988065+0000","last_peered":"2024-09-15T21:06:48.988065+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:48.988065+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5695","reported_seq":5773,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:46.605183+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:46.605183+0000","last_peered":"2024-09-15T21:06:46.605183+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:46.605183+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46626060,"num_objects":5692,"num_object_clones":0,"num_object_copies":22768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5693,"num_write_kb":45541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3411}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5684","reported_seq":5748,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.470042+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:48.470042+0000","last_peered":"2024-09-15T21:06:48.470042+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:48.470042+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46519564,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":746309288,"num_objects":91107,"num_object_clones":0,"num_object_copies":364428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91134,"num_write_kb":728954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1347637248,"data_stored":1347637248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91161,"ondisk_log_size":91161,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689480,"kb_used_data":364856,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93682360,"statfs":{"total":96636764160,"available":95930736640,"internally_reserved":0,"allocated":373612544,"data_stored":373455594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":686408,"kb_used_data":362296,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93685432,"statfs":{"total":96636764160,"available":95933882368,"internally_reserved":0,"allocated":370991104,"data_stored":370834154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447540,"kb_used_data":222948,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93924300,"statfs":{"total":96636764160,"available":96178483200,"internally_reserved":0,"allocated":228298752,"data_stored":228141802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698064,"kb_used_data":363772,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93673776,"statfs":{"total":96636764160,"available":95921946624,"internally_reserved":0,"allocated":372502528,"data_stored":372345578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373260288,"data_stored":373260288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228876288,"data_stored":228876288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":371433472,"data_stored":371433472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374067200,"data_stored":374067200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:51.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:52.386 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:52.387 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:52.407 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-15T21:06:51.988725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749119144,"num_objects":91450,"num_object_clones":0,"num_object_copies":365800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91477,"num_write_kb":731698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91504,"ondisk_log_size":91504,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2531080,"kb_used_data":1319300,"kb_used_omap":14,"kb_used_meta":1211697,"kb_avail":374956280,"statfs":{"total":386547056640,"available":383955230720,"internally_reserved":0,"allocated":1350963200,"data_stored":1350335400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1240778290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,12,19,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25034752,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004656"},"pg_stats":[{"pgid":"1.f","version":"31'5639","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.164971+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:49.164971+0000","last_peered":"2024-09-15T21:06:49.164971+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:49.164971+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5714","reported_seq":5775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.226528+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:48.226528+0000","last_peered":"2024-09-15T21:06:48.226528+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:48.226528+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773516,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5712,"num_write_kb":45692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5667","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.740510+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:51.740510+0000","last_peered":"2024-09-15T21:06:51.740510+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:51.740510+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46398086,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3406}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5677","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.360185+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:48.360185+0000","last_peered":"2024-09-15T21:06:48.360185+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:48.360185+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470412,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5803","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.935433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:48.935433+0000","last_peered":"2024-09-15T21:06:48.935433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:48.935433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47510796,"num_objects":5800,"num_object_clones":0,"num_object_copies":23200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5658","reported_seq":5697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.159053+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:49.159053+0000","last_peered":"2024-09-15T21:06:49.159053+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:49.159053+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46350336,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5832","reported_seq":5897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.355753+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:48.355753+0000","last_peered":"2024-09-15T21:06:48.355753+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:48.355753+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3460}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5810","reported_seq":5874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.375265+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:48.375265+0000","last_peered":"2024-09-15T21:06:48.375265+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:48.375265+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47551756,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5808,"num_write_kb":46452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5678","reported_seq":5733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.035206+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:49.035206+0000","last_peered":"2024-09-15T21:06:49.035206+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:49.035206+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46443032,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5808","reported_seq":5884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.751913+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:51.751913+0000","last_peered":"2024-09-15T21:06:51.751913+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:51.751913+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47559948,"num_objects":5806,"num_object_clones":0,"num_object_copies":23224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5665","reported_seq":9920,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.158300+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:49.158300+0000","last_peered":"2024-09-15T21:06:49.158300+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:49.158300+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46362514,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2103,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5660","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.085632+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:49.085632+0000","last_peered":"2024-09-15T21:06:49.085632+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:49.085632+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46357126,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5758","reported_seq":5833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.654973+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:51.654973+0000","last_peered":"2024-09-15T21:06:51.654973+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:51.654973+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159942,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5757,"num_write_kb":46055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5676","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.988065+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:48.988065+0000","last_peered":"2024-09-15T21:06:48.988065+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:48.988065+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5775","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.580841+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:51.580841+0000","last_peered":"2024-09-15T21:06:51.580841+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:51.580841+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5684","reported_seq":5748,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.470042+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:48.470042+0000","last_peered":"2024-09-15T21:06:48.470042+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:48.470042+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46519564,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749119144,"num_objects":91450,"num_object_clones":0,"num_object_copies":365800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91477,"num_write_kb":731698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1353170944,"data_stored":1353170944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91504,"ondisk_log_size":91504,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689480,"kb_used_data":364856,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93682360,"statfs":{"total":96636764160,"available":95930736640,"internally_reserved":0,"allocated":373612544,"data_stored":373455594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695996,"kb_used_data":367724,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93675844,"statfs":{"total":96636764160,"available":95924064256,"internally_reserved":0,"allocated":376549376,"data_stored":376392426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447540,"kb_used_data":222948,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93924300,"statfs":{"total":96636764160,"available":96178483200,"internally_reserved":0,"allocated":228298752,"data_stored":228141802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698064,"kb_used_data":363772,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93673776,"statfs":{"total":96636764160,"available":95921946624,"internally_reserved":0,"allocated":372502528,"data_stored":372345578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373260288,"data_stored":373260288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228876288,"data_stored":228876288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376967168,"data_stored":376967168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374067200,"data_stored":374067200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:52.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:52.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5639", 'reported_seq': 5680, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:49.164971+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:49.164971+0000', 'last_peered': '2024-09-15T21:06:49.164971+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:49.164971+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5639, 'log_dups_size': 0, 'ondisk_log_size': 5639, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46194688, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 22556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5639, 'num_write_kb': 45112, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3363}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:52.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5714", 'reported_seq': 5775, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.226528+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:48.226528+0000', 'last_peered': '2024-09-15T21:06:48.226528+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:48.226528+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5714, 'log_dups_size': 0, 'ondisk_log_size': 5714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46773516, 'num_objects': 5710, 'num_object_clones': 0, 'num_object_copies': 22840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5712, 'num_write_kb': 45692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:52.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5667", 'reported_seq': 5741, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:51.740510+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:51.740510+0000', 'last_peered': '2024-09-15T21:06:51.740510+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:51.740510+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5667, 'log_dups_size': 0, 'ondisk_log_size': 5667, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46398086, 'num_objects': 5664, 'num_object_clones': 0, 'num_object_copies': 22656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5666, 'num_write_kb': 45318, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3406}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5677", 'reported_seq': 5741, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.360185+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:48.360185+0000', 'last_peered': '2024-09-15T21:06:48.360185+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:48.360185+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5677, 'log_dups_size': 0, 'ondisk_log_size': 5677, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46470412, 'num_objects': 5673, 'num_object_clones': 0, 'num_object_copies': 22692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5675, 'num_write_kb': 45389, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5803", 'reported_seq': 5854, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.935433+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:48.935433+0000', 'last_peered': '2024-09-15T21:06:48.935433+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:48.935433+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5803, 'log_dups_size': 0, 'ondisk_log_size': 5803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47510796, 'num_objects': 5800, 'num_object_clones': 0, 'num_object_copies': 23200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5801, 'num_write_kb': 46405, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3489}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5658", 'reported_seq': 5697, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:49.159053+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:49.159053+0000', 'last_peered': '2024-09-15T21:06:49.159053+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:49.159053+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5658, 'log_dups_size': 0, 'ondisk_log_size': 5658, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46350336, 'num_objects': 5658, 'num_object_clones': 0, 'num_object_copies': 22632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5658, 'num_write_kb': 45264, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3342}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:52.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5832", 'reported_seq': 5897, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.355753+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:48.355753+0000', 'last_peered': '2024-09-15T21:06:48.355753+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:48.355753+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5832, 'log_dups_size': 0, 'ondisk_log_size': 5832, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47723788, 'num_objects': 5826, 'num_object_clones': 0, 'num_object_copies': 23304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5830, 'num_write_kb': 46620, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3460}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5810", 'reported_seq': 5874, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.375265+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:48.375265+0000', 'last_peered': '2024-09-15T21:06:48.375265+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:48.375265+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5810, 'log_dups_size': 0, 'ondisk_log_size': 5810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47551756, 'num_objects': 5805, 'num_object_clones': 0, 'num_object_copies': 23220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5808, 'num_write_kb': 46452, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3331}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5678", 'reported_seq': 5733, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:49.035206+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:49.035206+0000', 'last_peered': '2024-09-15T21:06:49.035206+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:49.035206+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5678, 'log_dups_size': 0, 'ondisk_log_size': 5678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46443032, 'num_objects': 5670, 'num_object_clones': 0, 'num_object_copies': 22680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5675, 'num_write_kb': 45377, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3338}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5808", 'reported_seq': 5884, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:51.751913+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:51.751913+0000', 'last_peered': '2024-09-15T21:06:51.751913+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:51.751913+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5808, 'log_dups_size': 0, 'ondisk_log_size': 5808, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47559948, 'num_objects': 5806, 'num_object_clones': 0, 'num_object_copies': 23224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5806, 'num_write_kb': 46446, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3471}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5665", 'reported_seq': 9920, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:49.158300+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:49.158300+0000', 'last_peered': '2024-09-15T21:06:49.158300+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:49.158300+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5665, 'log_dups_size': 0, 'ondisk_log_size': 5665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46362514, 'num_objects': 5660, 'num_object_clones': 0, 'num_object_copies': 22640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 163, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5662, 'num_write_kb': 45284, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2103, 'num_bytes_recovered': 17178624, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:52.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5660", 'reported_seq': 5695, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:49.085632+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:49.085632+0000', 'last_peered': '2024-09-15T21:06:49.085632+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:49.085632+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5660, 'log_dups_size': 0, 'ondisk_log_size': 5660, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46357126, 'num_objects': 5659, 'num_object_clones': 0, 'num_object_copies': 22636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5659, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5659, 'num_write_kb': 45271, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3403}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5758", 'reported_seq': 5833, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:51.654973+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:51.654973+0000', 'last_peered': '2024-09-15T21:06:51.654973+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:51.654973+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5758, 'log_dups_size': 0, 'ondisk_log_size': 5758, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47159942, 'num_objects': 5757, 'num_object_clones': 0, 'num_object_copies': 23028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5757, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5757, 'num_write_kb': 46055, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5676", 'reported_seq': 5728, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.988065+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:48.988065+0000', 'last_peered': '2024-09-15T21:06:48.988065+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:48.988065+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5676, 'log_dups_size': 0, 'ondisk_log_size': 5676, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46462220, 'num_objects': 5672, 'num_object_clones': 0, 'num_object_copies': 22688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5674, 'num_write_kb': 45388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3393}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5775", 'reported_seq': 5853, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:51.580841+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:51.580841+0000', 'last_peered': '2024-09-15T21:06:51.580841+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:51.580841+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5775, 'log_dups_size': 0, 'ondisk_log_size': 5775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47281420, 'num_objects': 5772, 'num_object_clones': 0, 'num_object_copies': 23088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5772, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5773, 'num_write_kb': 46181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3491}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:52.411 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5684", 'reported_seq': 5748, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:48.470042+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:48.470042+0000', 'last_peered': '2024-09-15T21:06:48.470042+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:48.470042+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5684, 'log_dups_size': 0, 'ondisk_log_size': 5684, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46519564, 'num_objects': 5679, 'num_object_clones': 0, 'num_object_copies': 22716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5679, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5682, 'num_write_kb': 45444, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3364}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:52.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:52.660 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 209 20 92255 92235 3.44722 1.88281 0.0801692 0.0453136 2024-09-15T21:06:53.154 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:53.155 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34986/365800 objects degraded (9.564%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":278,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91450,"data_bytes":749119144,"bytes_used":2591825920,"bytes_avail":383955230720,"bytes_total":386547056640,"degraded_objects":34986,"degraded_total":365800,"degraded_ratio":0.095642427556041548,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":91441,"recovering_keys_per_sec":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"write_bytes_sec":2085420,"read_op_per_sec":0,"write_op_per_sec":254},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:53.184 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:53.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:53.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 210 20 92502 92482 3.43998 1.92969 0.0605257 0.045408 2024-09-15T21:06:53.777 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:53.778 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:53.803 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-15T21:06:51.988725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":749119144,"num_objects":91450,"num_object_clones":0,"num_object_copies":365800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91477,"num_write_kb":731698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91504,"ondisk_log_size":91504,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2531080,"kb_used_data":1319300,"kb_used_omap":14,"kb_used_meta":1211697,"kb_avail":374956280,"statfs":{"total":386547056640,"available":383955230720,"internally_reserved":0,"allocated":1350963200,"data_stored":1350335400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1240778290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,12,19,17,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25034752,"num_objects":3056,"num_object_clones":0,"num_object_copies":12224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004656"},"pg_stats":[{"pgid":"1.f","version":"31'5639","reported_seq":5680,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.164971+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:49.164971+0000","last_peered":"2024-09-15T21:06:49.164971+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:49.164971+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5639,"log_dups_size":0,"ondisk_log_size":5639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46194688,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5639,"num_write_kb":45112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3363}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5714","reported_seq":5775,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.226528+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:48.226528+0000","last_peered":"2024-09-15T21:06:48.226528+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:48.226528+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46773516,"num_objects":5710,"num_object_clones":0,"num_object_copies":22840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5712,"num_write_kb":45692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5667","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.740510+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:51.740510+0000","last_peered":"2024-09-15T21:06:51.740510+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:51.740510+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46398086,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3406}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5677","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.360185+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:48.360185+0000","last_peered":"2024-09-15T21:06:48.360185+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:48.360185+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46470412,"num_objects":5673,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5803","reported_seq":5854,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.935433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:48.935433+0000","last_peered":"2024-09-15T21:06:48.935433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:48.935433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47510796,"num_objects":5800,"num_object_clones":0,"num_object_copies":23200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5801,"num_write_kb":46405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5658","reported_seq":5697,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.159053+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:49.159053+0000","last_peered":"2024-09-15T21:06:49.159053+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:49.159053+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5658,"log_dups_size":0,"ondisk_log_size":5658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46350336,"num_objects":5658,"num_object_clones":0,"num_object_copies":22632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5658,"num_write_kb":45264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3342}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5832","reported_seq":5897,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.355753+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:48.355753+0000","last_peered":"2024-09-15T21:06:48.355753+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:48.355753+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5832,"log_dups_size":0,"ondisk_log_size":5832,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47723788,"num_objects":5826,"num_object_clones":0,"num_object_copies":23304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":46620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3460}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5810","reported_seq":5874,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.375265+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:48.375265+0000","last_peered":"2024-09-15T21:06:48.375265+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:48.375265+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47551756,"num_objects":5805,"num_object_clones":0,"num_object_copies":23220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5808,"num_write_kb":46452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3331}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5678","reported_seq":5733,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.035206+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:49.035206+0000","last_peered":"2024-09-15T21:06:49.035206+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:49.035206+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46443032,"num_objects":5670,"num_object_clones":0,"num_object_copies":22680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5675,"num_write_kb":45377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5808","reported_seq":5884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.751913+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:51.751913+0000","last_peered":"2024-09-15T21:06:51.751913+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:51.751913+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47559948,"num_objects":5806,"num_object_clones":0,"num_object_copies":23224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5665","reported_seq":9920,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.158300+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:49.158300+0000","last_peered":"2024-09-15T21:06:49.158300+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:49.158300+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46362514,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5662,"num_write_kb":45284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2103,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5660","reported_seq":5695,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:49.085632+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:49.085632+0000","last_peered":"2024-09-15T21:06:49.085632+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:49.085632+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5660,"log_dups_size":0,"ondisk_log_size":5660,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46357126,"num_objects":5659,"num_object_clones":0,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":45271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3403}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5758","reported_seq":5833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.654973+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:51.654973+0000","last_peered":"2024-09-15T21:06:51.654973+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:51.654973+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159942,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5757,"num_write_kb":46055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5676","reported_seq":5728,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.988065+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:48.988065+0000","last_peered":"2024-09-15T21:06:48.988065+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:48.988065+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":45388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3393}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5775","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.580841+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:51.580841+0000","last_peered":"2024-09-15T21:06:51.580841+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:51.580841+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5684","reported_seq":5748,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:48.470042+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:48.470042+0000","last_peered":"2024-09-15T21:06:48.470042+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:48.470042+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5684,"log_dups_size":0,"ondisk_log_size":5684,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46519564,"num_objects":5679,"num_object_clones":0,"num_object_copies":22716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5682,"num_write_kb":45444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3364}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":749119144,"num_objects":91450,"num_object_clones":0,"num_object_copies":365800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91477,"num_write_kb":731698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2127,"num_bytes_recovered":17178624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1353170944,"data_stored":1353170944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91504,"ondisk_log_size":91504,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689480,"kb_used_data":364856,"kb_used_omap":4,"kb_used_meta":324603,"kb_avail":93682360,"statfs":{"total":96636764160,"available":95930736640,"internally_reserved":0,"allocated":373612544,"data_stored":373455594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":332394192},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,10,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695996,"kb_used_data":367724,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93675844,"statfs":{"total":96636764160,"available":95924064256,"internally_reserved":0,"allocated":376549376,"data_stored":376392426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051619,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":447540,"kb_used_data":222948,"kb_used_omap":4,"kb_used_meta":224571,"kb_avail":93924300,"statfs":{"total":96636764160,"available":96178483200,"internally_reserved":0,"allocated":228298752,"data_stored":228141802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":229961423},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698064,"kb_used_data":363772,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93673776,"statfs":{"total":96636764160,"available":95921946624,"internally_reserved":0,"allocated":372502528,"data_stored":372345578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":373260288,"data_stored":373260288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228876288,"data_stored":228876288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376967168,"data_stored":376967168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374067200,"data_stored":374067200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:54.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 211 20 92779 92759 3.43394 2.16406 0.0691581 0.0454889 2024-09-15T21:06:55.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 212 20 93054 93034 3.42787 2.14844 0.0518487 0.0455731 2024-09-15T21:06:56.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 213 20 93325 93305 3.42172 2.11719 0.0836019 0.04565 2024-09-15T21:06:56.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:57.418 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:57.418 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:57.450 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-15T21:06:55.990087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757261992,"num_objects":92444,"num_object_clones":0,"num_object_copies":369776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92471,"num_write_kb":739650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92498,"ondisk_log_size":92498,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2562356,"kb_used_data":1336692,"kb_used_omap":14,"kb_used_meta":1225585,"kb_avail":374925004,"statfs":{"total":386547056640,"available":383923204096,"internally_reserved":0,"allocated":1368772608,"data_stored":1368144808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1254999602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,9,5,16,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26730496,"num_objects":3263,"num_object_clones":0,"num_object_copies":13052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-213,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3263,"num_write_kb":26104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":1744896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004092"},"pg_stats":[{"pgid":"1.f","version":"31'5725","reported_seq":5766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.075035+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:54.075035+0000","last_peered":"2024-09-15T21:06:54.075035+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:54.075035+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46899200,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5725,"num_write_kb":45800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5789","reported_seq":5852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.131902+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:53.131902+0000","last_peered":"2024-09-15T21:06:53.131902+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:53.131902+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47387916,"num_objects":5785,"num_object_clones":0,"num_object_copies":23140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5667","reported_seq":5741,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.740510+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:51.740510+0000","last_peered":"2024-09-15T21:06:51.740510+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:51.740510+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5667,"log_dups_size":0,"ondisk_log_size":5667,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46398086,"num_objects":5664,"num_object_clones":0,"num_object_copies":22656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5666,"num_write_kb":45318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3406}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5750","reported_seq":5814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.189192+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:53.189192+0000","last_peered":"2024-09-15T21:06:53.189192+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:53.189192+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5900","reported_seq":5951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.087170+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:54.087170+0000","last_peered":"2024-09-15T21:06:54.087170+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:54.087170+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":47181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5727","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.131305+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:54.131305+0000","last_peered":"2024-09-15T21:06:54.131305+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:54.131305+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46915584,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5922","reported_seq":5987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.461453+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:53.461453+0000","last_peered":"2024-09-15T21:06:53.461453+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:53.461453+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3550}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5902","reported_seq":5966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.520944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:53.520944+0000","last_peered":"2024-09-15T21:06:53.520944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:53.520944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":47188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5762","reported_seq":5817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.066276+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:54.066276+0000","last_peered":"2024-09-15T21:06:54.066276+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:54.066276+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47131160,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5808","reported_seq":5884,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.751913+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:51.751913+0000","last_peered":"2024-09-15T21:06:51.751913+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:51.751913+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5808,"log_dups_size":0,"ondisk_log_size":5808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47559948,"num_objects":5806,"num_object_clones":0,"num_object_copies":23224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":46446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5737","reported_seq":10151,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.228133+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:54.228133+0000","last_peered":"2024-09-15T21:06:54.228133+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:54.228133+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46952338,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":45860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2182,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5755","reported_seq":5790,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.167656+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:54.167656+0000","last_peered":"2024-09-15T21:06:54.167656+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:54.167656+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47135366,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5758","reported_seq":5833,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.654973+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:51.654973+0000","last_peered":"2024-09-15T21:06:51.654973+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:51.654973+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47159942,"num_objects":5757,"num_object_clones":0,"num_object_copies":23028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5757,"num_write_kb":46055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3471}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5755","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.161828+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:54.161828+0000","last_peered":"2024-09-15T21:06:54.161828+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:54.161828+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47109388,"num_objects":5751,"num_object_clones":0,"num_object_copies":23004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":46020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5775","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:51.580841+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:51.580841+0000","last_peered":"2024-09-15T21:06:51.580841+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:51.580841+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47281420,"num_objects":5772,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5773,"num_write_kb":46181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3491}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5766","reported_seq":5830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.385439+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:53.385439+0000","last_peered":"2024-09-15T21:06:53.385439+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:53.385439+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757261992,"num_objects":92444,"num_object_clones":0,"num_object_copies":369776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92471,"num_write_kb":739650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369964544,"data_stored":1369964544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92498,"ondisk_log_size":92498,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698940,"kb_used_data":370156,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93672900,"statfs":{"total":96636764160,"available":95921049600,"internally_reserved":0,"allocated":379039744,"data_stored":378882794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695996,"kb_used_data":367724,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":93675844,"statfs":{"total":96636764160,"available":95924064256,"internally_reserved":0,"allocated":376549376,"data_stored":376392426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051620,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":458908,"kb_used_data":228748,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93912932,"statfs":{"total":96636764160,"available":96166842368,"internally_reserved":0,"allocated":234237952,"data_stored":234081002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,0,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708512,"kb_used_data":370064,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93663328,"statfs":{"total":96636764160,"available":95911247872,"internally_reserved":0,"allocated":378945536,"data_stored":378788586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378781696,"data_stored":378781696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":234708992,"data_stored":234708992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":376967168,"data_stored":376967168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379506688,"data_stored":379506688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:57.451 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:57.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 214 20 93578 93558 3.41496 1.97656 0.12727 0.0457406 2024-09-15T21:06:58.017 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:58.017 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:58.042 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-15T21:06:57.990793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760161960,"num_objects":92798,"num_object_clones":0,"num_object_copies":371192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92825,"num_write_kb":742482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92852,"ondisk_log_size":92852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564788,"kb_used_data":1347444,"kb_used_omap":14,"kb_used_meta":1217265,"kb_avail":374922572,"statfs":{"total":386547056640,"available":383920713728,"internally_reserved":0,"allocated":1379782656,"data_stored":1379154856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1246479922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,6,9,17,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24911872,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004026"},"pg_stats":[{"pgid":"1.f","version":"31'5725","reported_seq":5766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.075035+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:54.075035+0000","last_peered":"2024-09-15T21:06:54.075035+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:54.075035+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46899200,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5725,"num_write_kb":45800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5789","reported_seq":5852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.131902+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:53.131902+0000","last_peered":"2024-09-15T21:06:53.131902+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:53.131902+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47387916,"num_objects":5785,"num_object_clones":0,"num_object_copies":23140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5757","reported_seq":5828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.659294+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:56.659294+0000","last_peered":"2024-09-15T21:06:56.659294+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:56.659294+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47135366,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":46038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5750","reported_seq":5814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.189192+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:53.189192+0000","last_peered":"2024-09-15T21:06:53.189192+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:53.189192+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5900","reported_seq":5951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.087170+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:54.087170+0000","last_peered":"2024-09-15T21:06:54.087170+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:54.087170+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":47181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5727","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.131305+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:54.131305+0000","last_peered":"2024-09-15T21:06:54.131305+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:54.131305+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46915584,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5922","reported_seq":5987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.461453+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:53.461453+0000","last_peered":"2024-09-15T21:06:53.461453+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:53.461453+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3550}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5902","reported_seq":5966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.520944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:53.520944+0000","last_peered":"2024-09-15T21:06:53.520944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:53.520944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":47188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5762","reported_seq":5817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.066276+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:54.066276+0000","last_peered":"2024-09-15T21:06:54.066276+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:54.066276+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47131160,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5886","reported_seq":5963,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.658934+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:56.658934+0000","last_peered":"2024-09-15T21:06:56.658934+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:56.658934+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48198924,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":47070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5737","reported_seq":10151,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.228133+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:54.228133+0000","last_peered":"2024-09-15T21:06:54.228133+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:54.228133+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46952338,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":45860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2182,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5755","reported_seq":5790,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.167656+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:54.167656+0000","last_peered":"2024-09-15T21:06:54.167656+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:54.167656+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47135366,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5840","reported_seq":5915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.665018+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:56.665018+0000","last_peered":"2024-09-15T21:06:56.665018+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:56.665018+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5840,"log_dups_size":0,"ondisk_log_size":5840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47831686,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5839,"num_write_kb":46711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5755","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.161828+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:54.161828+0000","last_peered":"2024-09-15T21:06:54.161828+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:54.161828+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47109388,"num_objects":5751,"num_object_clones":0,"num_object_copies":23004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":46020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5879","reported_seq":5957,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.750205+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:56.750205+0000","last_peered":"2024-09-15T21:06:56.750205+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:56.750205+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5879,"log_dups_size":0,"ondisk_log_size":5879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48133388,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":47013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5766","reported_seq":5830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.385439+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:53.385439+0000","last_peered":"2024-09-15T21:06:53.385439+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:53.385439+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":760161960,"num_objects":92798,"num_object_clones":0,"num_object_copies":371192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92825,"num_write_kb":742482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1381163008,"data_stored":1381163008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92852,"ondisk_log_size":92852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698940,"kb_used_data":370156,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93672900,"statfs":{"total":96636764160,"available":95921049600,"internally_reserved":0,"allocated":379039744,"data_stored":378882794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688692,"kb_used_data":372900,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93683148,"statfs":{"total":96636764160,"available":95931543552,"internally_reserved":0,"allocated":381849600,"data_stored":381692650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468644,"kb_used_data":234324,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93903196,"statfs":{"total":96636764160,"available":96156872704,"internally_reserved":0,"allocated":239947776,"data_stored":239790826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708512,"kb_used_data":370064,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93663328,"statfs":{"total":96636764160,"available":95911247872,"internally_reserved":0,"allocated":378945536,"data_stored":378788586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378781696,"data_stored":378781696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240492544,"data_stored":240492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382382080,"data_stored":382382080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379506688,"data_stored":379506688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5725", 'reported_seq': 5766, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.075035+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:54.075035+0000', 'last_peered': '2024-09-15T21:06:54.075035+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:54.075035+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5725, 'log_dups_size': 0, 'ondisk_log_size': 5725, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46899200, 'num_objects': 5725, 'num_object_clones': 0, 'num_object_copies': 22900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5725, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5725, 'num_write_kb': 45800, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3449}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5789", 'reported_seq': 5852, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:53.131902+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:53.131902+0000', 'last_peered': '2024-09-15T21:06:53.131902+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:53.131902+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5789, 'log_dups_size': 0, 'ondisk_log_size': 5789, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47387916, 'num_objects': 5785, 'num_object_clones': 0, 'num_object_copies': 23140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5785, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5787, 'num_write_kb': 46292, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3405}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "31'5757", 'reported_seq': 5828, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:56.659294+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:06:56.659294+0000', 'last_peered': '2024-09-15T21:06:56.659294+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:06:56.659294+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5757, 'log_dups_size': 0, 'ondisk_log_size': 5757, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47135366, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5756, 'num_write_kb': 46038, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3496}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:06:58.043 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5750", 'reported_seq': 5814, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:53.189192+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:53.189192+0000', 'last_peered': '2024-09-15T21:06:53.189192+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:53.189192+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5750, 'log_dups_size': 0, 'ondisk_log_size': 5750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47068428, 'num_objects': 5746, 'num_object_clones': 0, 'num_object_copies': 22984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5746, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5748, 'num_write_kb': 45973, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3412}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5900", 'reported_seq': 5951, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.087170+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:54.087170+0000', 'last_peered': '2024-09-15T21:06:54.087170+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:54.087170+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5900, 'log_dups_size': 0, 'ondisk_log_size': 5900, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48305420, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5898, 'num_write_kb': 47181, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3586}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5727", 'reported_seq': 5767, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.131305+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:54.131305+0000', 'last_peered': '2024-09-15T21:06:54.131305+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:54.131305+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5727, 'log_dups_size': 0, 'ondisk_log_size': 5727, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46915584, 'num_objects': 5727, 'num_object_clones': 0, 'num_object_copies': 22908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5727, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5727, 'num_write_kb': 45816, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3411}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'5922", 'reported_seq': 5987, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:53.461453+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:53.461453+0000', 'last_peered': '2024-09-15T21:06:53.461453+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:53.461453+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5922, 'log_dups_size': 0, 'ondisk_log_size': 5922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48461068, 'num_objects': 5916, 'num_object_clones': 0, 'num_object_copies': 23664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5920, 'num_write_kb': 47340, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3550}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5902", 'reported_seq': 5966, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:53.520944+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:53.520944+0000', 'last_peered': '2024-09-15T21:06:53.520944+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:53.520944+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5902, 'log_dups_size': 0, 'ondisk_log_size': 5902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48305420, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5900, 'num_write_kb': 47188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5762", 'reported_seq': 5817, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.066276+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:54.066276+0000', 'last_peered': '2024-09-15T21:06:54.066276+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:54.066276+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5762, 'log_dups_size': 0, 'ondisk_log_size': 5762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47131160, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5759, 'num_write_kb': 46049, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3422}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:06:58.044 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "31'5886", 'reported_seq': 5963, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:56.658934+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:06:56.658934+0000', 'last_peered': '2024-09-15T21:06:56.658934+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:06:56.658934+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5886, 'log_dups_size': 0, 'ondisk_log_size': 5886, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48198924, 'num_objects': 5884, 'num_object_clones': 0, 'num_object_copies': 23536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5884, 'num_write_kb': 47070, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3549}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5737", 'reported_seq': 10151, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.228133+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:54.228133+0000', 'last_peered': '2024-09-15T21:06:54.228133+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:54.228133+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5737, 'log_dups_size': 0, 'ondisk_log_size': 5737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46952338, 'num_objects': 5732, 'num_object_clones': 0, 'num_object_copies': 22928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 84, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5732, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5734, 'num_write_kb': 45860, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2182, 'num_bytes_recovered': 17825792, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5648}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5755", 'reported_seq': 5790, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.167656+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:54.167656+0000', 'last_peered': '2024-09-15T21:06:54.167656+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:54.167656+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5755, 'log_dups_size': 0, 'ondisk_log_size': 5755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47135366, 'num_objects': 5754, 'num_object_clones': 0, 'num_object_copies': 23016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5754, 'num_write_kb': 46031, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "31'5840", 'reported_seq': 5915, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:56.665018+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:06:56.665018+0000', 'last_peered': '2024-09-15T21:06:56.665018+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:06:56.665018+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5840, 'log_dups_size': 0, 'ondisk_log_size': 5840, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47831686, 'num_objects': 5839, 'num_object_clones': 0, 'num_object_copies': 23356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5839, 'num_write_kb': 46711, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5755", 'reported_seq': 5806, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:54.161828+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:54.161828+0000', 'last_peered': '2024-09-15T21:06:54.161828+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:54.161828+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5755, 'log_dups_size': 0, 'ondisk_log_size': 5755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47109388, 'num_objects': 5751, 'num_object_clones': 0, 'num_object_copies': 23004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5753, 'num_write_kb': 46020, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:06:58.045 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "31'5879", 'reported_seq': 5957, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:56.750205+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:06:56.750205+0000', 'last_peered': '2024-09-15T21:06:56.750205+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:06:56.750205+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5879, 'log_dups_size': 0, 'ondisk_log_size': 5879, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48133388, 'num_objects': 5876, 'num_object_clones': 0, 'num_object_copies': 23504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5876, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5877, 'num_write_kb': 47013, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:06:58.046 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:06:58.046 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5766", 'reported_seq': 5830, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:53.385439+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:53.385439+0000', 'last_peered': '2024-09-15T21:06:53.385439+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:53.385439+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5766, 'log_dups_size': 0, 'ondisk_log_size': 5766, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47191308, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5764, 'num_write_kb': 46100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:06:58.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:06:58.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 215 20 93809 93789 3.40747 1.80469 0.0914635 0.0458406 2024-09-15T21:06:58.767 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:58.768 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34907/371192 objects degraded (9.404%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":284,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":92798,"data_bytes":760161960,"bytes_used":2626342912,"bytes_avail":383920713728,"bytes_total":386547056640,"degraded_objects":34907,"degraded_total":371192,"degraded_ratio":0.094040281040539672,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":98271,"recovering_keys_per_sec":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"write_bytes_sec":2075292,"read_op_per_sec":0,"write_op_per_sec":253},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:06:58.801 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:06:58.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:06:59.367 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:06:59.371 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:06:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-09-15T21:06:57.990793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":760161960,"num_objects":92798,"num_object_clones":0,"num_object_copies":371192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92825,"num_write_kb":742482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92852,"ondisk_log_size":92852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2564788,"kb_used_data":1347444,"kb_used_omap":14,"kb_used_meta":1217265,"kb_avail":374922572,"statfs":{"total":386547056640,"available":383920713728,"internally_reserved":0,"allocated":1379782656,"data_stored":1379154856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1246479922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2,6,9,17,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24911872,"num_objects":3041,"num_object_clones":0,"num_object_copies":12164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3041,"num_write_kb":24328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":1179648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004026"},"pg_stats":[{"pgid":"1.f","version":"31'5725","reported_seq":5766,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.075035+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:54.075035+0000","last_peered":"2024-09-15T21:06:54.075035+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:54.075035+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5725,"log_dups_size":0,"ondisk_log_size":5725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46899200,"num_objects":5725,"num_object_clones":0,"num_object_copies":22900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5725,"num_write_kb":45800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3449}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5789","reported_seq":5852,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.131902+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:53.131902+0000","last_peered":"2024-09-15T21:06:53.131902+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:53.131902+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5789,"log_dups_size":0,"ondisk_log_size":5789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47387916,"num_objects":5785,"num_object_clones":0,"num_object_copies":23140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5787,"num_write_kb":46292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3405}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"31'5757","reported_seq":5828,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.659294+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:06:56.659294+0000","last_peered":"2024-09-15T21:06:56.659294+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:06:56.659294+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5757,"log_dups_size":0,"ondisk_log_size":5757,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47135366,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5756,"num_write_kb":46038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3496}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5750","reported_seq":5814,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.189192+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:53.189192+0000","last_peered":"2024-09-15T21:06:53.189192+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:53.189192+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47068428,"num_objects":5746,"num_object_clones":0,"num_object_copies":22984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5748,"num_write_kb":45973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3412}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5900","reported_seq":5951,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.087170+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:54.087170+0000","last_peered":"2024-09-15T21:06:54.087170+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:54.087170+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5900,"log_dups_size":0,"ondisk_log_size":5900,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5898,"num_write_kb":47181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3586}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5727","reported_seq":5767,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.131305+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:54.131305+0000","last_peered":"2024-09-15T21:06:54.131305+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:54.131305+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5727,"log_dups_size":0,"ondisk_log_size":5727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46915584,"num_objects":5727,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5727,"num_write_kb":45816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'5922","reported_seq":5987,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.461453+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:53.461453+0000","last_peered":"2024-09-15T21:06:53.461453+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:53.461453+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5922,"log_dups_size":0,"ondisk_log_size":5922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48461068,"num_objects":5916,"num_object_clones":0,"num_object_copies":23664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5920,"num_write_kb":47340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3550}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5902","reported_seq":5966,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.520944+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:53.520944+0000","last_peered":"2024-09-15T21:06:53.520944+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:53.520944+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5902,"log_dups_size":0,"ondisk_log_size":5902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48305420,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5900,"num_write_kb":47188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3423}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5762","reported_seq":5817,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.066276+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:54.066276+0000","last_peered":"2024-09-15T21:06:54.066276+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:54.066276+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5762,"log_dups_size":0,"ondisk_log_size":5762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47131160,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":46049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"31'5886","reported_seq":5963,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.658934+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:06:56.658934+0000","last_peered":"2024-09-15T21:06:56.658934+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:06:56.658934+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48198924,"num_objects":5884,"num_object_clones":0,"num_object_copies":23536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":47070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3549}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5737","reported_seq":10151,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.228133+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:54.228133+0000","last_peered":"2024-09-15T21:06:54.228133+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:54.228133+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5737,"log_dups_size":0,"ondisk_log_size":5737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46952338,"num_objects":5732,"num_object_clones":0,"num_object_copies":22928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5734,"num_write_kb":45860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2182,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5755","reported_seq":5790,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.167656+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:54.167656+0000","last_peered":"2024-09-15T21:06:54.167656+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:54.167656+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47135366,"num_objects":5754,"num_object_clones":0,"num_object_copies":23016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":46031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"31'5840","reported_seq":5915,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.665018+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:06:56.665018+0000","last_peered":"2024-09-15T21:06:56.665018+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:06:56.665018+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5840,"log_dups_size":0,"ondisk_log_size":5840,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47831686,"num_objects":5839,"num_object_clones":0,"num_object_copies":23356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5839,"num_write_kb":46711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5755","reported_seq":5806,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:54.161828+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:54.161828+0000","last_peered":"2024-09-15T21:06:54.161828+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:54.161828+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5755,"log_dups_size":0,"ondisk_log_size":5755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47109388,"num_objects":5751,"num_object_clones":0,"num_object_copies":23004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5753,"num_write_kb":46020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3472}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"31'5879","reported_seq":5957,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:56.750205+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:06:56.750205+0000","last_peered":"2024-09-15T21:06:56.750205+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:06:56.750205+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5879,"log_dups_size":0,"ondisk_log_size":5879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48133388,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5877,"num_write_kb":47013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5766","reported_seq":5830,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:53.385439+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:53.385439+0000","last_peered":"2024-09-15T21:06:53.385439+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:53.385439+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5766,"log_dups_size":0,"ondisk_log_size":5766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5764,"num_write_kb":46100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":760161960,"num_objects":92798,"num_object_clones":0,"num_object_copies":371192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34907,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92825,"num_write_kb":742482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2206,"num_bytes_recovered":17825792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1381163008,"data_stored":1381163008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92852,"ondisk_log_size":92852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698940,"kb_used_data":370156,"kb_used_omap":4,"kb_used_meta":328763,"kb_avail":93672900,"statfs":{"total":96636764160,"available":95921049600,"internally_reserved":0,"allocated":379039744,"data_stored":378882794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":336654032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688692,"kb_used_data":372900,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":93683148,"statfs":{"total":96636764160,"available":95931543552,"internally_reserved":0,"allocated":381849600,"data_stored":381692650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,4,3,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468644,"kb_used_data":234324,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93903196,"statfs":{"total":96636764160,"available":96156872704,"internally_reserved":0,"allocated":239947776,"data_stored":239790826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708512,"kb_used_data":370064,"kb_used_omap":4,"kb_used_meta":338427,"kb_avail":93663328,"statfs":{"total":96636764160,"available":95911247872,"internally_reserved":0,"allocated":378945536,"data_stored":378788586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":346549966},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378781696,"data_stored":378781696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240492544,"data_stored":240492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":382382080,"data_stored":382382080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":379506688,"data_stored":379506688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:06:59.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 216 20 94066 94046 3.40099 2.00781 0.0481346 0.0459311 2024-09-15T21:07:00.661 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 217 20 94335 94315 3.39501 2.10156 0.0446463 0.0460077 2024-09-15T21:07:01.662 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 218 20 94591 94571 3.38861 2 0.0960911 0.0460994 2024-09-15T21:07:02.403 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:02.662 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 219 20 94846 94826 3.38223 1.99219 0.0566099 0.0461842 2024-09-15T21:07:03.000 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:03.001 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:03.023 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-15T21:07:01.992035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":770967208,"num_objects":94117,"num_object_clones":0,"num_object_copies":376468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94144,"num_write_kb":753034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2279,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94171,"ondisk_log_size":94171,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576212,"kb_used_data":1363020,"kb_used_omap":14,"kb_used_meta":1213105,"kb_avail":374911148,"statfs":{"total":386547056640,"available":383909015552,"internally_reserved":0,"allocated":1395732480,"data_stored":1395100671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1242220082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,9,2,5,9,15,21,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18472960,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002544"},"pg_stats":[{"pgid":"1.f","version":"31'5799","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.142764+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:59.142764+0000","last_peered":"2024-09-15T21:06:59.142764+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:59.142764+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47505408,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5876","reported_seq":5939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.584886+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:58.584886+0000","last_peered":"2024-09-15T21:06:58.584886+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:58.584886+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48100620,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5874,"num_write_kb":46988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5851","reported_seq":5928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.838903+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:01.838903+0000","last_peered":"2024-09-15T21:07:01.838903+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:01.838903+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47905414,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":46790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5839","reported_seq":5902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.618747+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:58.618747+0000","last_peered":"2024-09-15T21:06:58.618747+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:58.618747+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47797516,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5978","reported_seq":6029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.242307+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:59.242307+0000","last_peered":"2024-09-15T21:06:59.242307+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:59.242307+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48944396,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5813","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.270199+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:59.270199+0000","last_peered":"2024-09-15T21:06:59.270199+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:59.270199+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47620096,"num_objects":5813,"num_object_clones":0,"num_object_copies":23252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6012","reported_seq":6077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.547773+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:58.547773+0000","last_peered":"2024-09-15T21:06:58.547773+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:58.547773+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49198348,"num_objects":6006,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6010,"num_write_kb":48060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5972","reported_seq":6034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.662291+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:58.662291+0000","last_peered":"2024-09-15T21:06:58.662291+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:58.662291+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48878860,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5970,"num_write_kb":47748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5843","reported_seq":5898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.210531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:59.210531+0000","last_peered":"2024-09-15T21:06:59.210531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:59.210531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47794712,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5971","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.831131+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:01.831131+0000","last_peered":"2024-09-15T21:07:01.831131+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:01.831131+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48895244,"num_objects":5969,"num_object_clones":0,"num_object_copies":23876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5814","reported_seq":10374,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.303291+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:59.303291+0000","last_peered":"2024-09-15T21:06:59.303291+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:59.303291+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583122,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":46476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5824","reported_seq":5859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.221450+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:59.221450+0000","last_peered":"2024-09-15T21:06:59.221450+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:59.221450+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47700614,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":46583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5916","reported_seq":5995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.806689+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:01.806689+0000","last_peered":"2024-09-15T21:07:01.806689+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:01.806689+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48454278,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5836","reported_seq":5888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.232082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:59.232082+0000","last_peered":"2024-09-15T21:06:59.232082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:59.232082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5974","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.682211+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:01.682211+0000","last_peered":"2024-09-15T21:07:01.682211+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:01.682211+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48911628,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5853","reported_seq":5917,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.584881+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:58.584881+0000","last_peered":"2024-09-15T21:06:58.584881+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:58.584881+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47904012,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":770967208,"num_objects":94117,"num_object_clones":0,"num_object_copies":376468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94144,"num_write_kb":753034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2279,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1397280768,"data_stored":1397280768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94171,"ondisk_log_size":94171,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691692,"kb_used_data":375388,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93680148,"statfs":{"total":96636764160,"available":95928471552,"internally_reserved":0,"allocated":384397312,"data_stored":384240362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697928,"kb_used_data":377968,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93673912,"statfs":{"total":96636764160,"available":95922085888,"internally_reserved":0,"allocated":387039232,"data_stored":386878273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468644,"kb_used_data":234324,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93903196,"statfs":{"total":96636764160,"available":96156872704,"internally_reserved":0,"allocated":239947776,"data_stored":239790826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717948,"kb_used_data":375340,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93653892,"statfs":{"total":96636764160,"available":95901585408,"internally_reserved":0,"allocated":384348160,"data_stored":384191210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":350809806},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,1,2,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384262144,"data_stored":384262144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240492544,"data_stored":240492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387624960,"data_stored":387624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384901120,"data_stored":384901120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:03.024 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:03.653 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:03.654 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:03.663 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:07:03.661759+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0462792 2024-09-15T21:07:03.663 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:07:03.664 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 220 20 95082 95062 3.37521 1.84375 0.0502844 0.0462792 2024-09-15T21:07:03.677 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-09-15T21:07:01.992035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":770967208,"num_objects":94117,"num_object_clones":0,"num_object_copies":376468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94144,"num_write_kb":753034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2279,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94171,"ondisk_log_size":94171,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2576212,"kb_used_data":1363020,"kb_used_omap":14,"kb_used_meta":1213105,"kb_avail":374911148,"statfs":{"total":386547056640,"available":383909015552,"internally_reserved":0,"allocated":1395732480,"data_stored":1395100671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1242220082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,9,2,5,9,15,21,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18472960,"num_objects":2255,"num_object_clones":0,"num_object_copies":9020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2255,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002544"},"pg_stats":[{"pgid":"1.f","version":"31'5799","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.142764+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:59.142764+0000","last_peered":"2024-09-15T21:06:59.142764+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:59.142764+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47505408,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"31'5876","reported_seq":5939,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.584886+0000","last_change":"2024-09-15T21:03:59.617745+0000","last_active":"2024-09-15T21:06:58.584886+0000","last_peered":"2024-09-15T21:06:58.584886+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:06:58.584886+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48100620,"num_objects":5872,"num_object_clones":0,"num_object_copies":23488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5874,"num_write_kb":46988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3492}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5851","reported_seq":5928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.838903+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:01.838903+0000","last_peered":"2024-09-15T21:07:01.838903+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:01.838903+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47905414,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":46790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"31'5839","reported_seq":5902,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.618747+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:06:58.618747+0000","last_peered":"2024-09-15T21:06:58.618747+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:06:58.618747+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47797516,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5837,"num_write_kb":46685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5978","reported_seq":6029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.242307+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:59.242307+0000","last_peered":"2024-09-15T21:06:59.242307+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:59.242307+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48944396,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5813","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.270199+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:59.270199+0000","last_peered":"2024-09-15T21:06:59.270199+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:59.270199+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47620096,"num_objects":5813,"num_object_clones":0,"num_object_copies":23252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"31'6012","reported_seq":6077,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.547773+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:06:58.547773+0000","last_peered":"2024-09-15T21:06:58.547773+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:06:58.547773+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49198348,"num_objects":6006,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6010,"num_write_kb":48060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3640}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"31'5972","reported_seq":6034,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.662291+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:06:58.662291+0000","last_peered":"2024-09-15T21:06:58.662291+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:06:58.662291+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5972,"log_dups_size":0,"ondisk_log_size":5972,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48878860,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5970,"num_write_kb":47748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3493}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5843","reported_seq":5898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.210531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:59.210531+0000","last_peered":"2024-09-15T21:06:59.210531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:59.210531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47794712,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5971","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.831131+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:01.831131+0000","last_peered":"2024-09-15T21:07:01.831131+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:01.831131+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48895244,"num_objects":5969,"num_object_clones":0,"num_object_copies":23876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5814","reported_seq":10374,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.303291+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:59.303291+0000","last_peered":"2024-09-15T21:06:59.303291+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:59.303291+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583122,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":46476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5824","reported_seq":5859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.221450+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:59.221450+0000","last_peered":"2024-09-15T21:06:59.221450+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:59.221450+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47700614,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":46583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5916","reported_seq":5995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.806689+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:01.806689+0000","last_peered":"2024-09-15T21:07:01.806689+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:01.806689+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48454278,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5836","reported_seq":5888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.232082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:59.232082+0000","last_peered":"2024-09-15T21:06:59.232082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:59.232082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5974","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.682211+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:01.682211+0000","last_peered":"2024-09-15T21:07:01.682211+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:01.682211+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48911628,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"31'5853","reported_seq":5917,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:58.584881+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:06:58.584881+0000","last_peered":"2024-09-15T21:06:58.584881+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:06:58.584881+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5853,"log_dups_size":0,"ondisk_log_size":5853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47904012,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5851,"num_write_kb":46796,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3533}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":770967208,"num_objects":94117,"num_object_clones":0,"num_object_copies":376468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94144,"num_write_kb":753034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2279,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1397280768,"data_stored":1397280768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94171,"ondisk_log_size":94171,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691692,"kb_used_data":375388,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93680148,"statfs":{"total":96636764160,"available":95928471552,"internally_reserved":0,"allocated":384397312,"data_stored":384240362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697928,"kb_used_data":377968,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93673912,"statfs":{"total":96636764160,"available":95922085888,"internally_reserved":0,"allocated":387039232,"data_stored":386878273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051621,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":468644,"kb_used_data":234324,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93903196,"statfs":{"total":96636764160,"available":96156872704,"internally_reserved":0,"allocated":239947776,"data_stored":239790826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717948,"kb_used_data":375340,"kb_used_omap":4,"kb_used_meta":342587,"kb_avail":93653892,"statfs":{"total":96636764160,"available":95901585408,"internally_reserved":0,"allocated":384348160,"data_stored":384191210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":350809806},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,0,1,2,3,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384262144,"data_stored":384262144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":240492544,"data_stored":240492544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387624960,"data_stored":387624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384901120,"data_stored":384901120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "31'5799", 'reported_seq': 5840, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.142764+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:06:59.142764+0000', 'last_peered': '2024-09-15T21:06:59.142764+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:06:59.142764+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5799, 'log_dups_size': 0, 'ondisk_log_size': 5799, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47505408, 'num_objects': 5799, 'num_object_clones': 0, 'num_object_copies': 23196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5799, 'num_write_kb': 46392, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3523}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "31'5876", 'reported_seq': 5939, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:58.584886+0000', 'last_change': '2024-09-15T21:03:59.617745+0000', 'last_active': '2024-09-15T21:06:58.584886+0000', 'last_peered': '2024-09-15T21:06:58.584886+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:06:58.584886+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5876, 'log_dups_size': 0, 'ondisk_log_size': 5876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48100620, 'num_objects': 5872, 'num_object_clones': 0, 'num_object_copies': 23488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2382, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5874, 'num_write_kb': 46988, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5851", 'reported_seq': 5928, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:01.838903+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:01.838903+0000', 'last_peered': '2024-09-15T21:07:01.838903+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:01.838903+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5851, 'log_dups_size': 0, 'ondisk_log_size': 5851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47905414, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5850, 'num_write_kb': 46790, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:03.678 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "31'5839", 'reported_seq': 5902, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:58.618747+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:06:58.618747+0000', 'last_peered': '2024-09-15T21:06:58.618747+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:06:58.618747+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5839, 'log_dups_size': 0, 'ondisk_log_size': 5839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47797516, 'num_objects': 5835, 'num_object_clones': 0, 'num_object_copies': 23340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5837, 'num_write_kb': 46685, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "31'5978", 'reported_seq': 6029, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.242307+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:06:59.242307+0000', 'last_peered': '2024-09-15T21:06:59.242307+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:06:59.242307+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5978, 'log_dups_size': 0, 'ondisk_log_size': 5978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48944396, 'num_objects': 5975, 'num_object_clones': 0, 'num_object_copies': 23900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5975, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5976, 'num_write_kb': 47805, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3664}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "31'5813", 'reported_seq': 5853, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.270199+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:06:59.270199+0000', 'last_peered': '2024-09-15T21:06:59.270199+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:06:59.270199+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5813, 'log_dups_size': 0, 'ondisk_log_size': 5813, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47620096, 'num_objects': 5813, 'num_object_clones': 0, 'num_object_copies': 23252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5813, 'num_write_kb': 46504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3497}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "31'6012", 'reported_seq': 6077, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:58.547773+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:06:58.547773+0000', 'last_peered': '2024-09-15T21:06:58.547773+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:06:58.547773+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6012, 'log_dups_size': 0, 'ondisk_log_size': 6012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49198348, 'num_objects': 6006, 'num_object_clones': 0, 'num_object_copies': 24024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6010, 'num_write_kb': 48060, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3640}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "31'5972", 'reported_seq': 6034, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:58.662291+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:06:58.662291+0000', 'last_peered': '2024-09-15T21:06:58.662291+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:06:58.662291+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5972, 'log_dups_size': 0, 'ondisk_log_size': 5972, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48878860, 'num_objects': 5967, 'num_object_clones': 0, 'num_object_copies': 23868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5970, 'num_write_kb': 47748, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3493}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:03.679 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "31'5843", 'reported_seq': 5898, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.210531+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:06:59.210531+0000', 'last_peered': '2024-09-15T21:06:59.210531+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:06:59.210531+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5843, 'log_dups_size': 0, 'ondisk_log_size': 5843, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47794712, 'num_objects': 5835, 'num_object_clones': 0, 'num_object_copies': 23340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5840, 'num_write_kb': 46697, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3503}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'5971", 'reported_seq': 6050, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:01.831131+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:01.831131+0000', 'last_peered': '2024-09-15T21:07:01.831131+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:01.831131+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5971, 'log_dups_size': 0, 'ondisk_log_size': 5971, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48895244, 'num_objects': 5969, 'num_object_clones': 0, 'num_object_copies': 23876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5969, 'num_write_kb': 47750, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3634}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "31'5814", 'reported_seq': 10374, 'reported_epoch': 31, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.303291+0000', 'last_change': '2024-09-15T21:03:59.618892+0000', 'last_active': '2024-09-15T21:06:59.303291+0000', 'last_peered': '2024-09-15T21:06:59.303291+0000', 'last_clean': '2024-09-15T21:02:50.620551+0000', 'last_became_active': '2024-09-15T21:03:59.607148+0000', 'last_became_peered': '2024-09-15T21:03:59.607148+0000', 'last_unstale': '2024-09-15T21:06:59.303291+0000', 'last_undegraded': '2024-09-15T21:03:59.115601+0000', 'last_fullsized': '2024-09-15T21:03:59.042793+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5814, 'log_dups_size': 0, 'ondisk_log_size': 5814, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47583122, 'num_objects': 5809, 'num_object_clones': 0, 'num_object_copies': 23236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 11, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5809, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5811, 'num_write_kb': 46476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2255, 'num_bytes_recovered': 18423808, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 5798}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "31'5824", 'reported_seq': 5859, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.221450+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:06:59.221450+0000', 'last_peered': '2024-09-15T21:06:59.221450+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:06:59.221450+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5824, 'log_dups_size': 0, 'ondisk_log_size': 5824, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47700614, 'num_objects': 5823, 'num_object_clones': 0, 'num_object_copies': 23292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5823, 'num_write_kb': 46583, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3567}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5916", 'reported_seq': 5995, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:01.806689+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:01.806689+0000', 'last_peered': '2024-09-15T21:07:01.806689+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:01.806689+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5916, 'log_dups_size': 0, 'ondisk_log_size': 5916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48454278, 'num_objects': 5915, 'num_object_clones': 0, 'num_object_copies': 23660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5915, 'num_write_kb': 47319, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3629}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:03.680 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "31'5836", 'reported_seq': 5888, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:59.232082+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:06:59.232082+0000', 'last_peered': '2024-09-15T21:06:59.232082+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:06:59.232082+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5836, 'log_dups_size': 0, 'ondisk_log_size': 5836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47772940, 'num_objects': 5832, 'num_object_clones': 0, 'num_object_copies': 23328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5834, 'num_write_kb': 46668, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:03.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:03.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'5974", 'reported_seq': 6056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:01.682211+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:01.682211+0000', 'last_peered': '2024-09-15T21:07:01.682211+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:01.682211+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5974, 'log_dups_size': 0, 'ondisk_log_size': 5974, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48911628, 'num_objects': 5971, 'num_object_clones': 0, 'num_object_copies': 23884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5972, 'num_write_kb': 47773, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3690}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:03.681 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:03.681 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "31'5853", 'reported_seq': 5917, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:06:58.584881+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:06:58.584881+0000', 'last_peered': '2024-09-15T21:06:58.584881+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:06:58.584881+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5853, 'log_dups_size': 0, 'ondisk_log_size': 5853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47904012, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5851, 'num_write_kb': 46796, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3533}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:03.681 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:04.374 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:04.374 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34834/376468 objects degraded (9.253%), 16 pgs degraded, 16 pgs undersized","count":32},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":290,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94117,"data_bytes":770967208,"bytes_used":2638041088,"bytes_avail":383909015552,"bytes_total":386547056640,"degraded_objects":34834,"degraded_total":376468,"degraded_ratio":0.092528448633084348,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":155598,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":1245184,"num_keys_recovered":0,"write_bytes_sec":2308385,"read_op_per_sec":0,"write_op_per_sec":281},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:04.399 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:04.400 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:04.664 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 221 20 95309 95289 3.36796 1.77344 0.157213 0.0463756 2024-09-15T21:07:05.086 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:05.087 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:05.109 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-15T21:07:03.992856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":774391464,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2595020,"kb_used_data":1373500,"kb_used_omap":14,"kb_used_meta":1221425,"kb_avail":374892340,"statfs":{"total":386547056640,"available":383889756160,"internally_reserved":0,"allocated":1406464000,"data_stored":1405816113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1250739762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,6,9,15,20,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17129472,"num_objects":2091,"num_object_clones":0,"num_object_copies":8364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2091,"num_write_kb":16728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002769"},"pg_stats":[{"pgid":"1.f","version":"31'5799","reported_seq":5840,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.142764+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:06:59.142764+0000","last_peered":"2024-09-15T21:06:59.142764+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:06:59.142764+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5799,"log_dups_size":0,"ondisk_log_size":5799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47505408,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5799,"num_write_kb":46392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5964","reported_seq":6112,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.855964+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:03.855964+0000","last_peered":"2024-09-15T21:07:03.855964+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:03.855964+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821516,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5962,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5851","reported_seq":5928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.838903+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:01.838903+0000","last_peered":"2024-09-15T21:07:01.838903+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:01.838903+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5851,"log_dups_size":0,"ondisk_log_size":5851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47905414,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5850,"num_write_kb":46790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5912","reported_seq":5980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.727161+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:03.727161+0000","last_peered":"2024-09-15T21:07:03.727161+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:03.727161+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"31'5978","reported_seq":6029,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.242307+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:06:59.242307+0000","last_peered":"2024-09-15T21:06:59.242307+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:06:59.242307+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5978,"log_dups_size":0,"ondisk_log_size":5978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48944396,"num_objects":5975,"num_object_clones":0,"num_object_copies":23900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5976,"num_write_kb":47805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3664}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"31'5813","reported_seq":5853,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.270199+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:06:59.270199+0000","last_peered":"2024-09-15T21:06:59.270199+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:06:59.270199+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47620096,"num_objects":5813,"num_object_clones":0,"num_object_copies":23252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":46504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3497}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6093","reported_seq":6162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.836735+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:03.836735+0000","last_peered":"2024-09-15T21:07:03.836735+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:03.836735+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49861900,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6049","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.838302+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:03.838302+0000","last_peered":"2024-09-15T21:07:03.838302+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:03.838302+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49509644,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"31'5843","reported_seq":5898,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.210531+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:06:59.210531+0000","last_peered":"2024-09-15T21:06:59.210531+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:06:59.210531+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5843,"log_dups_size":0,"ondisk_log_size":5843,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47794712,"num_objects":5835,"num_object_clones":0,"num_object_copies":23340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5840,"num_write_kb":46697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3503}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'5971","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.831131+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:01.831131+0000","last_peered":"2024-09-15T21:07:01.831131+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:01.831131+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5971,"log_dups_size":0,"ondisk_log_size":5971,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48895244,"num_objects":5969,"num_object_clones":0,"num_object_copies":23876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":47750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3634}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"31'5814","reported_seq":10374,"reported_epoch":31,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.303291+0000","last_change":"2024-09-15T21:03:59.618892+0000","last_active":"2024-09-15T21:06:59.303291+0000","last_peered":"2024-09-15T21:06:59.303291+0000","last_clean":"2024-09-15T21:02:50.620551+0000","last_became_active":"2024-09-15T21:03:59.607148+0000","last_became_peered":"2024-09-15T21:03:59.607148+0000","last_unstale":"2024-09-15T21:06:59.303291+0000","last_undegraded":"2024-09-15T21:03:59.115601+0000","last_fullsized":"2024-09-15T21:03:59.042793+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47583122,"num_objects":5809,"num_object_clones":0,"num_object_copies":23236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5811,"num_write_kb":46476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":18423808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":5798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"31'5824","reported_seq":5859,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.221450+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:06:59.221450+0000","last_peered":"2024-09-15T21:06:59.221450+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:06:59.221450+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5824,"log_dups_size":0,"ondisk_log_size":5824,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47700614,"num_objects":5823,"num_object_clones":0,"num_object_copies":23292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":46583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3567}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5916","reported_seq":5995,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.806689+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:01.806689+0000","last_peered":"2024-09-15T21:07:01.806689+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:01.806689+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48454278,"num_objects":5915,"num_object_clones":0,"num_object_copies":23660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3629}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"31'5836","reported_seq":5888,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:06:59.232082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:06:59.232082+0000","last_peered":"2024-09-15T21:06:59.232082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:06:59.232082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5836,"log_dups_size":0,"ondisk_log_size":5836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5834,"num_write_kb":46668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3553}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'5974","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:01.682211+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:01.682211+0000","last_peered":"2024-09-15T21:07:01.682211+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:01.682211+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5974,"log_dups_size":0,"ondisk_log_size":5974,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48911628,"num_objects":5971,"num_object_clones":0,"num_object_copies":23884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5972,"num_write_kb":47773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3690}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5952","reported_seq":6020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.682322+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:03.682322+0000","last_peered":"2024-09-15T21:07:03.682322+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:03.682322+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48715020,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":774391464,"num_objects":94535,"num_object_clones":0,"num_object_copies":378140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34792,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94562,"num_write_kb":756378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":18751488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1408311296,"data_stored":1408311296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94589,"ondisk_log_size":94589,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691692,"kb_used_data":375388,"kb_used_omap":4,"kb_used_meta":316283,"kb_avail":93680148,"statfs":{"total":96636764160,"available":95928471552,"internally_reserved":0,"allocated":384397312,"data_stored":384240362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":323874512},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,4,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697928,"kb_used_data":377968,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93673912,"statfs":{"total":96636764160,"available":95922085888,"internally_reserved":0,"allocated":387039232,"data_stored":386878273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,3,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051622,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":478156,"kb_used_data":239672,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93893684,"statfs":{"total":96636764160,"available":96147132416,"internally_reserved":0,"allocated":245424128,"data_stored":245259139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727244,"kb_used_data":380472,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93644596,"statfs":{"total":96636764160,"available":95892066304,"internally_reserved":0,"allocated":389603328,"data_stored":389438339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389632000,"data_stored":389632000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246153216,"data_stored":246153216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":387624960,"data_stored":387624960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":384901120,"data_stored":384901120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:05.664 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 222 20 95563 95543 3.36173 1.98438 0.111385 0.0464669 2024-09-15T21:07:06.664 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 223 20 95825 95805 3.35583 2.04688 0.046414 0.0465471 2024-09-15T21:07:07.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 224 20 96075 96055 3.34951 1.95312 0.0711588 0.0466328 2024-09-15T21:07:08.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:08.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 225 20 96324 96304 3.34327 1.94531 0.147872 0.0467226 2024-09-15T21:07:08.714 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:08.714 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:08.738 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-15T21:07:07.994404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":781190824,"num_objects":95365,"num_object_clones":0,"num_object_copies":381460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95392,"num_write_kb":763018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2330,"num_bytes_recovered":18841600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95419,"ondisk_log_size":95419,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658876,"kb_used_data":1388792,"kb_used_omap":14,"kb_used_meta":1270001,"kb_avail":374828484,"statfs":{"total":386547056640,"available":383824367616,"internally_reserved":0,"allocated":1422123008,"data_stored":1421463052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1300481586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,6,9,19,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13090816,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-53,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1598,"num_write_kb":12784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":417792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003096"},"pg_stats":[{"pgid":"1.f","version":"33'5882","reported_seq":5927,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.207707+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:04.207707+0000","last_peered":"2024-09-15T21:07:04.207707+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:04.207707+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48185344,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5964","reported_seq":6112,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.855964+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:03.855964+0000","last_peered":"2024-09-15T21:07:03.855964+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:03.855964+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821516,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5962,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5916","reported_seq":5994,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.711410+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:06.711410+0000","last_peered":"2024-09-15T21:07:06.711410+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:06.711410+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48437894,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5912","reported_seq":5980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.727161+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:03.727161+0000","last_peered":"2024-09-15T21:07:03.727161+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:03.727161+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6054","reported_seq":6109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.276031+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:04.276031+0000","last_peered":"2024-09-15T21:07:04.276031+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:04.276031+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49566988,"num_objects":6051,"num_object_clones":0,"num_object_copies":24204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":48413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5886","reported_seq":5930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.707230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:03.707230+0000","last_peered":"2024-09-15T21:07:03.707230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:03.707230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":47088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6093","reported_seq":6162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.836735+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:03.836735+0000","last_peered":"2024-09-15T21:07:03.836735+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:03.836735+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49861900,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6049","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.838302+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:03.838302+0000","last_peered":"2024-09-15T21:07:03.838302+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:03.838302+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49509644,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5917","reported_seq":5976,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.232516+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:04.232516+0000","last_peered":"2024-09-15T21:07:04.232516+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:04.232516+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48400920,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6036","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.889137+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:06.889137+0000","last_peered":"2024-09-15T21:07:06.889137+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:06.889137+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49427724,"num_objects":6034,"num_object_clones":0,"num_object_copies":24136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5876","reported_seq":10495,"reported_epoch":33,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:07:04.248583+0000","last_change":"2024-09-15T21:07:01.606689+0000","last_active":"2024-09-15T21:07:04.248583+0000","last_peered":"2024-09-15T21:07:04.248583+0000","last_clean":"2024-09-15T21:07:04.248583+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:04.248583+0000","last_undegraded":"2024-09-15T21:07:04.248583+0000","last_fullsized":"2024-09-15T21:07:04.248583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":2049,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48091026,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":18513920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5912","reported_seq":5951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.144742+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:04.144742+0000","last_peered":"2024-09-15T21:07:04.144742+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:04.144742+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421510,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5911,"num_write_kb":47287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3655}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5982","reported_seq":6061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.798614+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:06.798614+0000","last_peered":"2024-09-15T21:07:06.798614+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:06.798614+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48994950,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5926","reported_seq":5981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.292924+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:04.292924+0000","last_peered":"2024-09-15T21:07:04.292924+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:04.292924+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":47388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6062","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.879618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:06.879618+0000","last_peered":"2024-09-15T21:07:06.879618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:06.879618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6062,"log_dups_size":0,"ondisk_log_size":6062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6060,"num_write_kb":48477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5952","reported_seq":6020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.682322+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:03.682322+0000","last_peered":"2024-09-15T21:07:03.682322+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:03.682322+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48715020,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":781190824,"num_objects":95365,"num_object_clones":0,"num_object_copies":381460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95392,"num_write_kb":763018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2330,"num_bytes_recovered":18841600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1423863808,"data_stored":1423863808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95419,"ondisk_log_size":95419,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719084,"kb_used_data":380572,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93652756,"statfs":{"total":96636764160,"available":95900422144,"internally_reserved":0,"allocated":389705728,"data_stored":389540739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725132,"kb_used_data":382972,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646708,"statfs":{"total":96636764160,"available":95894228992,"internally_reserved":0,"allocated":392163328,"data_stored":391998339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":244776,"kb_used_omap":4,"kb_used_meta":242619,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":250650624,"data_stored":250485635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":248442575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727244,"kb_used_data":380472,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93644596,"statfs":{"total":96636764160,"available":95892066304,"internally_reserved":0,"allocated":389603328,"data_stored":389438339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389632000,"data_stored":389632000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251404288,"data_stored":251404288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392765440,"data_stored":392765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":390062080,"data_stored":390062080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:08.740 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:09.403 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:09.403 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:09.429 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-15T21:07:07.994404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":781190824,"num_objects":95365,"num_object_clones":0,"num_object_copies":381460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95392,"num_write_kb":763018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2330,"num_bytes_recovered":18841600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95419,"ondisk_log_size":95419,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658876,"kb_used_data":1388792,"kb_used_omap":14,"kb_used_meta":1270001,"kb_avail":374828484,"statfs":{"total":386547056640,"available":383824367616,"internally_reserved":0,"allocated":1422123008,"data_stored":1421463052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1300481586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,6,9,19,12,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13090816,"num_objects":1598,"num_object_clones":0,"num_object_copies":6392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-53,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1598,"num_write_kb":12784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":417792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003096"},"pg_stats":[{"pgid":"1.f","version":"33'5882","reported_seq":5927,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.207707+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:04.207707+0000","last_peered":"2024-09-15T21:07:04.207707+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:04.207707+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48185344,"num_objects":5882,"num_object_clones":0,"num_object_copies":23528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5882,"num_write_kb":47056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3606}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'5964","reported_seq":6112,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.855964+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:03.855964+0000","last_peered":"2024-09-15T21:07:03.855964+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:03.855964+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5964,"log_dups_size":0,"ondisk_log_size":5964,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48821516,"num_objects":5960,"num_object_clones":0,"num_object_copies":23840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2340,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5962,"num_write_kb":47692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3620}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5916","reported_seq":5994,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.711410+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:06.711410+0000","last_peered":"2024-09-15T21:07:06.711410+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:06.711410+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48437894,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5912","reported_seq":5980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.727161+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:03.727161+0000","last_peered":"2024-09-15T21:07:03.727161+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:03.727161+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48395532,"num_objects":5908,"num_object_clones":0,"num_object_copies":23632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5910,"num_write_kb":47269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3574}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6054","reported_seq":6109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.276031+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:04.276031+0000","last_peered":"2024-09-15T21:07:04.276031+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:04.276031+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6054,"log_dups_size":0,"ondisk_log_size":6054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49566988,"num_objects":6051,"num_object_clones":0,"num_object_copies":24204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6052,"num_write_kb":48413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3740}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5886","reported_seq":5930,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.707230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:03.707230+0000","last_peered":"2024-09-15T21:07:03.707230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:03.707230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5886,"log_dups_size":0,"ondisk_log_size":5886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48218112,"num_objects":5886,"num_object_clones":0,"num_object_copies":23544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5886,"num_write_kb":47088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6093","reported_seq":6162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.836735+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:03.836735+0000","last_peered":"2024-09-15T21:07:03.836735+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:03.836735+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49861900,"num_objects":6087,"num_object_clones":0,"num_object_copies":24348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":48708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3721}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6049","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.838302+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:03.838302+0000","last_peered":"2024-09-15T21:07:03.838302+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:03.838302+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6049,"log_dups_size":0,"ondisk_log_size":6049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49509644,"num_objects":6044,"num_object_clones":0,"num_object_copies":24176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6047,"num_write_kb":48364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3570}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5917","reported_seq":5976,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.232516+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:04.232516+0000","last_peered":"2024-09-15T21:07:04.232516+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:04.232516+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5917,"log_dups_size":0,"ondisk_log_size":5917,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48400920,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5914,"num_write_kb":47289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6036","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.889137+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:06.889137+0000","last_peered":"2024-09-15T21:07:06.889137+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:06.889137+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49427724,"num_objects":6034,"num_object_clones":0,"num_object_copies":24136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5876","reported_seq":10495,"reported_epoch":33,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:07:04.248583+0000","last_change":"2024-09-15T21:07:01.606689+0000","last_active":"2024-09-15T21:07:04.248583+0000","last_peered":"2024-09-15T21:07:04.248583+0000","last_clean":"2024-09-15T21:07:04.248583+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:04.248583+0000","last_undegraded":"2024-09-15T21:07:04.248583+0000","last_fullsized":"2024-09-15T21:07:04.248583+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":2049,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48091026,"num_objects":5871,"num_object_clones":0,"num_object_copies":23484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5873,"num_write_kb":46972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":18513920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5912","reported_seq":5951,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.144742+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:04.144742+0000","last_peered":"2024-09-15T21:07:04.144742+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:04.144742+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48421510,"num_objects":5911,"num_object_clones":0,"num_object_copies":23644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5911,"num_write_kb":47287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3655}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5982","reported_seq":6061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.798614+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:06.798614+0000","last_peered":"2024-09-15T21:07:06.798614+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:06.798614+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48994950,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'5926","reported_seq":5981,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:04.292924+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:04.292924+0000","last_peered":"2024-09-15T21:07:04.292924+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:04.292924+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5926,"log_dups_size":0,"ondisk_log_size":5926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5924,"num_write_kb":47388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6062","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.879618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:06.879618+0000","last_peered":"2024-09-15T21:07:06.879618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:06.879618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6062,"log_dups_size":0,"ondisk_log_size":6062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6060,"num_write_kb":48477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'5952","reported_seq":6020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:03.682322+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:03.682322+0000","last_peered":"2024-09-15T21:07:03.682322+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:03.682322+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5952,"log_dups_size":0,"ondisk_log_size":5952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48715020,"num_objects":5947,"num_object_clones":0,"num_object_copies":23788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":47588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":781190824,"num_objects":95365,"num_object_clones":0,"num_object_copies":381460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34781,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95392,"num_write_kb":763018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2330,"num_bytes_recovered":18841600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1423863808,"data_stored":1423863808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95419,"ondisk_log_size":95419,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719084,"kb_used_data":380572,"kb_used_omap":4,"kb_used_meta":338491,"kb_avail":93652756,"statfs":{"total":96636764160,"available":95900422144,"internally_reserved":0,"allocated":389705728,"data_stored":389540739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":346615504},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,3,5,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725132,"kb_used_data":382972,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646708,"statfs":{"total":96636764160,"available":95894228992,"internally_reserved":0,"allocated":392163328,"data_stored":391998339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":244776,"kb_used_omap":4,"kb_used_meta":242619,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":250650624,"data_stored":250485635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":248442575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727244,"kb_used_data":380472,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93644596,"statfs":{"total":96636764160,"available":95892066304,"internally_reserved":0,"allocated":389603328,"data_stored":389438339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":355069646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389632000,"data_stored":389632000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251404288,"data_stored":251404288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392765440,"data_stored":392765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":390062080,"data_stored":390062080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:09.430 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:09.430 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5882", 'reported_seq': 5927, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:04.207707+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:04.207707+0000', 'last_peered': '2024-09-15T21:07:04.207707+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:04.207707+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5882, 'log_dups_size': 0, 'ondisk_log_size': 5882, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48185344, 'num_objects': 5882, 'num_object_clones': 0, 'num_object_copies': 23528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5882, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5882, 'num_write_kb': 47056, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3606}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'5964", 'reported_seq': 6112, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.855964+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:03.855964+0000', 'last_peered': '2024-09-15T21:07:03.855964+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:03.855964+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5964, 'log_dups_size': 0, 'ondisk_log_size': 5964, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48821516, 'num_objects': 5960, 'num_object_clones': 0, 'num_object_copies': 23840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2340, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5960, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5962, 'num_write_kb': 47692, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 42, 'num_bytes_recovered': 327680, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3620}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'5916", 'reported_seq': 5994, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:06.711410+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:06.711410+0000', 'last_peered': '2024-09-15T21:07:06.711410+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:06.711410+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5916, 'log_dups_size': 0, 'ondisk_log_size': 5916, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48437894, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5915, 'num_write_kb': 47310, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3655}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5912", 'reported_seq': 5980, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.727161+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:03.727161+0000', 'last_peered': '2024-09-15T21:07:03.727161+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:03.727161+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5912, 'log_dups_size': 0, 'ondisk_log_size': 5912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48395532, 'num_objects': 5908, 'num_object_clones': 0, 'num_object_copies': 23632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5908, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5910, 'num_write_kb': 47269, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6054", 'reported_seq': 6109, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:04.276031+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:04.276031+0000', 'last_peered': '2024-09-15T21:07:04.276031+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:04.276031+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6054, 'log_dups_size': 0, 'ondisk_log_size': 6054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49566988, 'num_objects': 6051, 'num_object_clones': 0, 'num_object_copies': 24204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6052, 'num_write_kb': 48413, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3740}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:09.431 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5886", 'reported_seq': 5930, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.707230+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:03.707230+0000', 'last_peered': '2024-09-15T21:07:03.707230+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:03.707230+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5886, 'log_dups_size': 0, 'ondisk_log_size': 5886, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48218112, 'num_objects': 5886, 'num_object_clones': 0, 'num_object_copies': 23544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5886, 'num_write_kb': 47088, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6093", 'reported_seq': 6162, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.836735+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:03.836735+0000', 'last_peered': '2024-09-15T21:07:03.836735+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:03.836735+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6093, 'log_dups_size': 0, 'ondisk_log_size': 6093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49861900, 'num_objects': 6087, 'num_object_clones': 0, 'num_object_copies': 24348, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6087, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6091, 'num_write_kb': 48708, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3721}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6049", 'reported_seq': 6117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.838302+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:03.838302+0000', 'last_peered': '2024-09-15T21:07:03.838302+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:03.838302+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6049, 'log_dups_size': 0, 'ondisk_log_size': 6049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49509644, 'num_objects': 6044, 'num_object_clones': 0, 'num_object_copies': 24176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6044, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6047, 'num_write_kb': 48364, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3570}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5917", 'reported_seq': 5976, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:04.232516+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:04.232516+0000', 'last_peered': '2024-09-15T21:07:04.232516+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:04.232516+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5917, 'log_dups_size': 0, 'ondisk_log_size': 5917, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48400920, 'num_objects': 5909, 'num_object_clones': 0, 'num_object_copies': 23636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5914, 'num_write_kb': 47289, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3577}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6036", 'reported_seq': 6117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:06.889137+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:06.889137+0000', 'last_peered': '2024-09-15T21:07:06.889137+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:06.889137+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6036, 'log_dups_size': 0, 'ondisk_log_size': 6036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49427724, 'num_objects': 6034, 'num_object_clones': 0, 'num_object_copies': 24136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6034, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6034, 'num_write_kb': 48270, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3699}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.6 is not active+clean 2024-09-15T21:07:09.432 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.6', 'version': "33'5876", 'reported_seq': 10495, 'reported_epoch': 33, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-15T21:07:04.248583+0000', 'last_change': '2024-09-15T21:07:01.606689+0000', 'last_active': '2024-09-15T21:07:04.248583+0000', 'last_peered': '2024-09-15T21:07:04.248583+0000', 'last_clean': '2024-09-15T21:07:04.248583+0000', 'last_became_active': '2024-09-15T21:07:01.480618+0000', 'last_became_peered': '2024-09-15T21:07:01.480618+0000', 'last_unstale': '2024-09-15T21:07:04.248583+0000', 'last_undegraded': '2024-09-15T21:07:04.248583+0000', 'last_fullsized': '2024-09-15T21:07:04.248583+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 33, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 2049, 'log_size': 5876, 'log_dups_size': 0, 'ondisk_log_size': 5876, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 2s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48091026, 'num_objects': 5871, 'num_object_clones': 0, 'num_object_copies': 23484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5873, 'num_write_kb': 46972, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2266, 'num_bytes_recovered': 18513920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5912", 'reported_seq': 5951, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:04.144742+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:04.144742+0000', 'last_peered': '2024-09-15T21:07:04.144742+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:04.144742+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5912, 'log_dups_size': 0, 'ondisk_log_size': 5912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48421510, 'num_objects': 5911, 'num_object_clones': 0, 'num_object_copies': 23644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5911, 'num_write_kb': 47287, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3655}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'5982", 'reported_seq': 6061, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:06.798614+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:06.798614+0000', 'last_peered': '2024-09-15T21:07:06.798614+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:06.798614+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5982, 'log_dups_size': 0, 'ondisk_log_size': 5982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48994950, 'num_objects': 5981, 'num_object_clones': 0, 'num_object_copies': 23924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5981, 'num_write_kb': 47847, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3695}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'5926", 'reported_seq': 5981, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:04.292924+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:04.292924+0000', 'last_peered': '2024-09-15T21:07:04.292924+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:04.292924+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5926, 'log_dups_size': 0, 'ondisk_log_size': 5926, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48510220, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 23688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5924, 'num_write_kb': 47388, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6062", 'reported_seq': 6143, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:06.879618+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:06.879618+0000', 'last_peered': '2024-09-15T21:07:06.879618+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:06.879618+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6062, 'log_dups_size': 0, 'ondisk_log_size': 6062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49632524, 'num_objects': 6059, 'num_object_clones': 0, 'num_object_copies': 24236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6060, 'num_write_kb': 48477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3778}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:09.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'5952", 'reported_seq': 6020, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:03.682322+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:03.682322+0000', 'last_peered': '2024-09-15T21:07:03.682322+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:03.682322+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5952, 'log_dups_size': 0, 'ondisk_log_size': 5952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48715020, 'num_objects': 5947, 'num_object_clones': 0, 'num_object_copies': 23788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5950, 'num_write_kb': 47588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:09.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:09.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 226 20 96565 96545 3.33681 1.88281 0.0922051 0.0468151 2024-09-15T21:07:10.193 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:10.194 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34781/381460 objects degraded (9.118%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":295,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":95365,"data_bytes":781190824,"bytes_used":2722689024,"bytes_avail":383824367616,"bytes_total":386547056640,"degraded_objects":34781,"degraded_total":381460,"degraded_ratio":0.091178629476222947,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":52203,"recovering_keys_per_sec":0,"num_objects_recovered":51,"num_bytes_recovered":417792,"num_keys_recovered":0,"write_bytes_sec":1635718,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:10.221 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:10.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:10.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 227 20 96806 96786 3.3304 1.88281 0.0624616 0.0469043 2024-09-15T21:07:10.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:10.831 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:10.857 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-15T21:07:09.995135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":788879126,"num_objects":96303,"num_object_clones":0,"num_object_copies":385212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90447,"num_write_kb":723470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96360,"ondisk_log_size":96360,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2660532,"kb_used_data":1398772,"kb_used_omap":14,"kb_used_meta":1261681,"kb_avail":374826828,"statfs":{"total":386547056640,"available":383822671872,"internally_reserved":0,"allocated":1432342528,"data_stored":1431682572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1291961906},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,10,5,15,12,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20779118,"num_objects":2536,"num_object_clones":0,"num_object_copies":10144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3347,"num_write_kb":-26764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2176,"num_bytes_recovered":-17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.593900"},"pg_stats":[{"pgid":"1.f","version":"33'5955","reported_seq":6000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.111828+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:09.111828+0000","last_peered":"2024-09-15T21:07:09.111828+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:09.111828+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48783360,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6043","reported_seq":6269,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.868184+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:08.868184+0000","last_peered":"2024-09-15T21:07:08.868184+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:08.868184+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468684,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'5916","reported_seq":5994,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.711410+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:06.711410+0000","last_peered":"2024-09-15T21:07:06.711410+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:06.711410+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5916,"log_dups_size":0,"ondisk_log_size":5916,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48437894,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5915,"num_write_kb":47310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3655}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5988","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.914057+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:08.914057+0000","last_peered":"2024-09-15T21:07:08.914057+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:08.914057+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49018124,"num_objects":5984,"num_object_clones":0,"num_object_copies":23936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6128","reported_seq":6183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.121363+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:09.121363+0000","last_peered":"2024-09-15T21:07:09.121363+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:09.121363+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50173196,"num_objects":6125,"num_object_clones":0,"num_object_copies":24500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":49005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5967","reported_seq":6011,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.239539+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:09.239539+0000","last_peered":"2024-09-15T21:07:09.239539+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:09.239539+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48881664,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":47736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6174","reported_seq":6243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.674892+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:08.674892+0000","last_peered":"2024-09-15T21:07:08.674892+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:08.674892+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3802}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6141","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.756781+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:08.756781+0000","last_peered":"2024-09-15T21:07:08.756781+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:08.756781+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50263308,"num_objects":6136,"num_object_clones":0,"num_object_copies":24544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":49100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5991","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.946556+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:08.946556+0000","last_peered":"2024-09-15T21:07:08.946556+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:08.946556+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49007128,"num_objects":5983,"num_object_clones":0,"num_object_copies":23932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6036","reported_seq":6117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.889137+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:06.889137+0000","last_peered":"2024-09-15T21:07:06.889137+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:06.889137+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6036,"log_dups_size":0,"ondisk_log_size":6036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49427724,"num_objects":6034,"num_object_clones":0,"num_object_copies":24136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6034,"num_write_kb":48270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3699}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5939","reported_seq":10602,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:09.168976+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:09.168976+0000","last_peered":"2024-09-15T21:07:09.168976+0000","last_clean":"2024-09-15T21:07:09.168976+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:09.168976+0000","last_undegraded":"2024-09-15T21:07:09.168976+0000","last_fullsized":"2024-09-15T21:07:09.168976+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":5939,"log_dups_size":0,"ondisk_log_size":5939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48586752,"num_objects":5931,"num_object_clones":0,"num_object_copies":23724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5996","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.306102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:09.306102+0000","last_peered":"2024-09-15T21:07:09.306102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:09.306102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5996,"log_dups_size":0,"ondisk_log_size":5996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49109638,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'5982","reported_seq":6061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.798614+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:06.798614+0000","last_peered":"2024-09-15T21:07:06.798614+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:06.798614+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5982,"log_dups_size":0,"ondisk_log_size":5982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48994950,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5981,"num_write_kb":47847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6009","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.269138+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:09.269138+0000","last_peered":"2024-09-15T21:07:09.269138+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:09.269138+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190156,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":48052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6062","reported_seq":6143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:06.879618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:06.879618+0000","last_peered":"2024-09-15T21:07:06.879618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:06.879618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6062,"log_dups_size":0,"ondisk_log_size":6062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49632524,"num_objects":6059,"num_object_clones":0,"num_object_copies":24236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6060,"num_write_kb":48477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3778}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6033","reported_seq":6101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.882668+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:08.882668+0000","last_peered":"2024-09-15T21:07:08.882668+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:08.882668+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49378572,"num_objects":6028,"num_object_clones":0,"num_object_copies":24112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":788879126,"num_objects":96303,"num_object_clones":0,"num_object_copies":385212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90447,"num_write_kb":723470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1434169344,"data_stored":1434169344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96360,"ondisk_log_size":96360,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728112,"kb_used_data":385440,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93643728,"statfs":{"total":96636764160,"available":95891177472,"internally_reserved":0,"allocated":394690560,"data_stored":394525571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725132,"kb_used_data":382972,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93646708,"statfs":{"total":96636764160,"available":95894228992,"internally_reserved":0,"allocated":392163328,"data_stored":391998339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,9,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051623,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":487416,"kb_used_data":244776,"kb_used_omap":4,"kb_used_meta":242619,"kb_avail":93884424,"statfs":{"total":96636764160,"available":96137650176,"internally_reserved":0,"allocated":250650624,"data_stored":250485635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":248442575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719872,"kb_used_data":385584,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93651968,"statfs":{"total":96636764160,"available":95899615232,"internally_reserved":0,"allocated":394838016,"data_stored":394673027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394805248,"data_stored":394805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251404288,"data_stored":251404288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":392765440,"data_stored":392765440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395194368,"data_stored":395194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:11.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 228 20 97070 97050 3.32484 2.0625 0.0736588 0.0469797 2024-09-15T21:07:12.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 229 20 97333 97313 3.3193 2.05469 0.0563847 0.0470584 2024-09-15T21:07:13.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 230 20 97610 97590 3.31427 2.16406 0.102902 0.0471297 2024-09-15T21:07:13.859 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:14.443 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:14.443 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:14.464 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-15T21:07:13.996431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791631638,"num_objects":96639,"num_object_clones":0,"num_object_copies":386556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90783,"num_write_kb":726158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96696,"ondisk_log_size":96696,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680268,"kb_used_data":1408780,"kb_used_omap":14,"kb_used_meta":1271409,"kb_avail":374807092,"statfs":{"total":386547056640,"available":383802462208,"internally_reserved":0,"allocated":1442590720,"data_stored":1441930764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1301923378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,13,4,7,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20664430,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3361,"num_write_kb":-26876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2176,"num_bytes_recovered":-17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004395"},"pg_stats":[{"pgid":"1.f","version":"33'5955","reported_seq":6000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.111828+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:09.111828+0000","last_peered":"2024-09-15T21:07:09.111828+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:09.111828+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48783360,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6043","reported_seq":6269,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.868184+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:08.868184+0000","last_peered":"2024-09-15T21:07:08.868184+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:08.868184+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468684,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6014","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.931373+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:11.931373+0000","last_peered":"2024-09-15T21:07:11.931373+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:11.931373+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240710,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":48094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3753}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5988","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.914057+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:08.914057+0000","last_peered":"2024-09-15T21:07:08.914057+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:08.914057+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49018124,"num_objects":5984,"num_object_clones":0,"num_object_copies":23936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6128","reported_seq":6183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.121363+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:09.121363+0000","last_peered":"2024-09-15T21:07:09.121363+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:09.121363+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50173196,"num_objects":6125,"num_object_clones":0,"num_object_copies":24500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":49005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5967","reported_seq":6011,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.239539+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:09.239539+0000","last_peered":"2024-09-15T21:07:09.239539+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:09.239539+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48881664,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":47736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6174","reported_seq":6243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.674892+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:08.674892+0000","last_peered":"2024-09-15T21:07:08.674892+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:08.674892+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3802}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6141","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.756781+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:08.756781+0000","last_peered":"2024-09-15T21:07:08.756781+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:08.756781+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50263308,"num_objects":6136,"num_object_clones":0,"num_object_copies":24544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":49100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5991","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.946556+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:08.946556+0000","last_peered":"2024-09-15T21:07:08.946556+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:08.946556+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49007128,"num_objects":5983,"num_object_clones":0,"num_object_copies":23932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6106","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.842635+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:11.842635+0000","last_peered":"2024-09-15T21:07:11.842635+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:11.842635+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6106,"log_dups_size":0,"ondisk_log_size":6106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001164,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5939","reported_seq":10602,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:09.168976+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:09.168976+0000","last_peered":"2024-09-15T21:07:09.168976+0000","last_clean":"2024-09-15T21:07:09.168976+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:09.168976+0000","last_undegraded":"2024-09-15T21:07:09.168976+0000","last_fullsized":"2024-09-15T21:07:09.168976+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":5939,"log_dups_size":0,"ondisk_log_size":5939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48586752,"num_objects":5931,"num_object_clones":0,"num_object_copies":23724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5996","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.306102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:09.306102+0000","last_peered":"2024-09-15T21:07:09.306102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:09.306102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5996,"log_dups_size":0,"ondisk_log_size":5996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49109638,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6072","reported_seq":6151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.852246+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:11.852246+0000","last_peered":"2024-09-15T21:07:11.852246+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:11.852246+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49732230,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6071,"num_write_kb":48567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3785}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6009","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.269138+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:09.269138+0000","last_peered":"2024-09-15T21:07:09.269138+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:09.269138+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190156,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":48052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6140","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.896541+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:11.896541+0000","last_peered":"2024-09-15T21:07:11.896541+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:11.896541+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6033","reported_seq":6101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.882668+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:08.882668+0000","last_peered":"2024-09-15T21:07:08.882668+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:08.882668+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49378572,"num_objects":6028,"num_object_clones":0,"num_object_copies":24112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791631638,"num_objects":96639,"num_object_clones":0,"num_object_copies":386556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90783,"num_write_kb":726158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1444655104,"data_stored":1444655104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96696,"ondisk_log_size":96696,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728112,"kb_used_data":385440,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93643728,"statfs":{"total":96636764160,"available":95891177472,"internally_reserved":0,"allocated":394690560,"data_stored":394525571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752196,"kb_used_data":387828,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93619644,"statfs":{"total":96636764160,"available":95866515456,"internally_reserved":0,"allocated":397135872,"data_stored":396970883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480088,"kb_used_data":249928,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93891752,"statfs":{"total":96636764160,"available":96145154048,"internally_reserved":0,"allocated":255926272,"data_stored":255761283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719872,"kb_used_data":385584,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93651968,"statfs":{"total":96636764160,"available":95899615232,"internally_reserved":0,"allocated":394838016,"data_stored":394673027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394805248,"data_stored":394805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256774144,"data_stored":256774144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397881344,"data_stored":397881344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395194368,"data_stored":395194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:14.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:14.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 231 20 97857 97837 3.30828 1.92969 0.13338 0.0472175 2024-09-15T21:07:15.083 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:15.084 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:15.105 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-15T21:07:13.996431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":791631638,"num_objects":96639,"num_object_clones":0,"num_object_copies":386556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90783,"num_write_kb":726158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96696,"ondisk_log_size":96696,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2680268,"kb_used_data":1408780,"kb_used_omap":14,"kb_used_meta":1271409,"kb_avail":374807092,"statfs":{"total":386547056640,"available":383802462208,"internally_reserved":0,"allocated":1442590720,"data_stored":1441930764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1301923378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,13,4,7,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20664430,"num_objects":2522,"num_object_clones":0,"num_object_copies":10088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-3361,"num_write_kb":-26876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2176,"num_bytes_recovered":-17776640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004395"},"pg_stats":[{"pgid":"1.f","version":"33'5955","reported_seq":6000,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.111828+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:09.111828+0000","last_peered":"2024-09-15T21:07:09.111828+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:09.111828+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5955,"log_dups_size":0,"ondisk_log_size":5955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48783360,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":47640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3679}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6043","reported_seq":6269,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.868184+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:08.868184+0000","last_peered":"2024-09-15T21:07:08.868184+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:08.868184+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49468684,"num_objects":6039,"num_object_clones":0,"num_object_copies":24156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6041,"num_write_kb":48324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3738}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6014","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.931373+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:11.931373+0000","last_peered":"2024-09-15T21:07:11.931373+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:11.931373+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240710,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":48094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3753}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'5988","reported_seq":6056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.914057+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:08.914057+0000","last_peered":"2024-09-15T21:07:08.914057+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:08.914057+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5988,"log_dups_size":0,"ondisk_log_size":5988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49018124,"num_objects":5984,"num_object_clones":0,"num_object_copies":23936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5986,"num_write_kb":47877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3650}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6128","reported_seq":6183,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.121363+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:09.121363+0000","last_peered":"2024-09-15T21:07:09.121363+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:09.121363+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6128,"log_dups_size":0,"ondisk_log_size":6128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50173196,"num_objects":6125,"num_object_clones":0,"num_object_copies":24500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6126,"num_write_kb":49005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'5967","reported_seq":6011,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.239539+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:09.239539+0000","last_peered":"2024-09-15T21:07:09.239539+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:09.239539+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48881664,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5967,"num_write_kb":47736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6174","reported_seq":6243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.674892+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:08.674892+0000","last_peered":"2024-09-15T21:07:08.674892+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:08.674892+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6174,"log_dups_size":0,"ondisk_log_size":6174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6172,"num_write_kb":49356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3802}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6141","reported_seq":6209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.756781+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:08.756781+0000","last_peered":"2024-09-15T21:07:08.756781+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:08.756781+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6141,"log_dups_size":0,"ondisk_log_size":6141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50263308,"num_objects":6136,"num_object_clones":0,"num_object_copies":24544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":49100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3662}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'5991","reported_seq":6050,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.946556+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:08.946556+0000","last_peered":"2024-09-15T21:07:08.946556+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:08.946556+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5991,"log_dups_size":0,"ondisk_log_size":5991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49007128,"num_objects":5983,"num_object_clones":0,"num_object_copies":23932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":47881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3651}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6106","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.842635+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:11.842635+0000","last_peered":"2024-09-15T21:07:11.842635+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:11.842635+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6106,"log_dups_size":0,"ondisk_log_size":6106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001164,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'5939","reported_seq":10602,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:09.168976+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:09.168976+0000","last_peered":"2024-09-15T21:07:09.168976+0000","last_clean":"2024-09-15T21:07:09.168976+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:09.168976+0000","last_undegraded":"2024-09-15T21:07:09.168976+0000","last_fullsized":"2024-09-15T21:07:09.168976+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":5939,"log_dups_size":0,"ondisk_log_size":5939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48586752,"num_objects":5931,"num_object_clones":0,"num_object_copies":23724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'5996","reported_seq":6034,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.306102+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:09.306102+0000","last_peered":"2024-09-15T21:07:09.306102+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:09.306102+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":5996,"log_dups_size":0,"ondisk_log_size":5996,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49109638,"num_objects":5995,"num_object_clones":0,"num_object_copies":23980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5995,"num_write_kb":47959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3739}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6072","reported_seq":6151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.852246+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:11.852246+0000","last_peered":"2024-09-15T21:07:11.852246+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:11.852246+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49732230,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6071,"num_write_kb":48567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3785}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6009","reported_seq":6063,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:09.269138+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:09.269138+0000","last_peered":"2024-09-15T21:07:09.269138+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:09.269138+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49190156,"num_objects":6005,"num_object_clones":0,"num_object_copies":24020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":48052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6140","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.896541+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:11.896541+0000","last_peered":"2024-09-15T21:07:11.896541+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:11.896541+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6033","reported_seq":6101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:08.882668+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:08.882668+0000","last_peered":"2024-09-15T21:07:08.882668+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:08.882668+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6033,"log_dups_size":0,"ondisk_log_size":6033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49378572,"num_objects":6028,"num_object_clones":0,"num_object_copies":24112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6031,"num_write_kb":48236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3713}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":791631638,"num_objects":96639,"num_object_clones":0,"num_object_copies":386556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90783,"num_write_kb":726158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":647168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1444655104,"data_stored":1444655104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96696,"ondisk_log_size":96696,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728112,"kb_used_data":385440,"kb_used_omap":4,"kb_used_meta":342651,"kb_avail":93643728,"statfs":{"total":96636764160,"available":95891177472,"internally_reserved":0,"allocated":394690560,"data_stored":394525571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":350875344},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,4,0,0,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752196,"kb_used_data":387828,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93619644,"statfs":{"total":96636764160,"available":95866515456,"internally_reserved":0,"allocated":397135872,"data_stored":396970883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480088,"kb_used_data":249928,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93891752,"statfs":{"total":96636764160,"available":96145154048,"internally_reserved":0,"allocated":255926272,"data_stored":255761283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719872,"kb_used_data":385584,"kb_used_omap":4,"kb_used_meta":334267,"kb_avail":93651968,"statfs":{"total":96636764160,"available":95899615232,"internally_reserved":0,"allocated":394838016,"data_stored":394673027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":342290126},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394805248,"data_stored":394805248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256774144,"data_stored":256774144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397881344,"data_stored":397881344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":395194368,"data_stored":395194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'5955", 'reported_seq': 6000, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:09.111828+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:09.111828+0000', 'last_peered': '2024-09-15T21:07:09.111828+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:09.111828+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5955, 'log_dups_size': 0, 'ondisk_log_size': 5955, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48783360, 'num_objects': 5955, 'num_object_clones': 0, 'num_object_copies': 23820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5955, 'num_write_kb': 47640, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3679}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6043", 'reported_seq': 6269, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.868184+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:08.868184+0000', 'last_peered': '2024-09-15T21:07:08.868184+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:08.868184+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6043, 'log_dups_size': 0, 'ondisk_log_size': 6043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49468684, 'num_objects': 6039, 'num_object_clones': 0, 'num_object_copies': 24156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2301, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6039, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6041, 'num_write_kb': 48324, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 81, 'num_bytes_recovered': 647168, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3738}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6014", 'reported_seq': 6090, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:11.931373+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:11.931373+0000', 'last_peered': '2024-09-15T21:07:11.931373+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:11.931373+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6014, 'log_dups_size': 0, 'ondisk_log_size': 6014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49240710, 'num_objects': 6011, 'num_object_clones': 0, 'num_object_copies': 24044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6013, 'num_write_kb': 48094, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3753}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:15.106 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'5988", 'reported_seq': 6056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.914057+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:08.914057+0000', 'last_peered': '2024-09-15T21:07:08.914057+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:08.914057+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5988, 'log_dups_size': 0, 'ondisk_log_size': 5988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49018124, 'num_objects': 5984, 'num_object_clones': 0, 'num_object_copies': 23936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5986, 'num_write_kb': 47877, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6128", 'reported_seq': 6183, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:09.121363+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:09.121363+0000', 'last_peered': '2024-09-15T21:07:09.121363+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:09.121363+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6128, 'log_dups_size': 0, 'ondisk_log_size': 6128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50173196, 'num_objects': 6125, 'num_object_clones': 0, 'num_object_copies': 24500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6125, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6126, 'num_write_kb': 49005, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3814}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'5967", 'reported_seq': 6011, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:09.239539+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:09.239539+0000', 'last_peered': '2024-09-15T21:07:09.239539+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:09.239539+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5967, 'log_dups_size': 0, 'ondisk_log_size': 5967, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48881664, 'num_objects': 5967, 'num_object_clones': 0, 'num_object_copies': 23868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5967, 'num_write_kb': 47736, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6174", 'reported_seq': 6243, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.674892+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:08.674892+0000', 'last_peered': '2024-09-15T21:07:08.674892+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:08.674892+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6174, 'log_dups_size': 0, 'ondisk_log_size': 6174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50525452, 'num_objects': 6168, 'num_object_clones': 0, 'num_object_copies': 24672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6172, 'num_write_kb': 49356, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3802}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6141", 'reported_seq': 6209, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.756781+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:08.756781+0000', 'last_peered': '2024-09-15T21:07:08.756781+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:08.756781+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6141, 'log_dups_size': 0, 'ondisk_log_size': 6141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50263308, 'num_objects': 6136, 'num_object_clones': 0, 'num_object_copies': 24544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6139, 'num_write_kb': 49100, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3662}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:15.107 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'5991", 'reported_seq': 6050, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.946556+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:08.946556+0000', 'last_peered': '2024-09-15T21:07:08.946556+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:08.946556+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5991, 'log_dups_size': 0, 'ondisk_log_size': 5991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49007128, 'num_objects': 5983, 'num_object_clones': 0, 'num_object_copies': 23932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5988, 'num_write_kb': 47881, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3651}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6106", 'reported_seq': 6186, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:11.842635+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:11.842635+0000', 'last_peered': '2024-09-15T21:07:11.842635+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:11.842635+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6106, 'log_dups_size': 0, 'ondisk_log_size': 6106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50001164, 'num_objects': 6104, 'num_object_clones': 0, 'num_object_copies': 24416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6104, 'num_write_kb': 48830, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3769}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'5996", 'reported_seq': 6034, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:09.306102+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:09.306102+0000', 'last_peered': '2024-09-15T21:07:09.306102+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:09.306102+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 5996, 'log_dups_size': 0, 'ondisk_log_size': 5996, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49109638, 'num_objects': 5995, 'num_object_clones': 0, 'num_object_copies': 23980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5995, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 5995, 'num_write_kb': 47959, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6072", 'reported_seq': 6151, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:11.852246+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:11.852246+0000', 'last_peered': '2024-09-15T21:07:11.852246+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:11.852246+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6072, 'log_dups_size': 0, 'ondisk_log_size': 6072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49732230, 'num_objects': 6071, 'num_object_clones': 0, 'num_object_copies': 24284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6071, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6071, 'num_write_kb': 48567, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3785}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:15.108 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:15.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6009", 'reported_seq': 6063, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:09.269138+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:09.269138+0000', 'last_peered': '2024-09-15T21:07:09.269138+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:09.269138+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6009, 'log_dups_size': 0, 'ondisk_log_size': 6009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49190156, 'num_objects': 6005, 'num_object_clones': 0, 'num_object_copies': 24020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6005, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6007, 'num_write_kb': 48052, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:15.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:15.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6140", 'reported_seq': 6221, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:11.896541+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:11.896541+0000', 'last_peered': '2024-09-15T21:07:11.896541+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:11.896541+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6140, 'log_dups_size': 0, 'ondisk_log_size': 6140, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50271500, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 24548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6138, 'num_write_kb': 49101, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3856}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:15.109 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:15.109 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6033", 'reported_seq': 6101, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:08.882668+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:08.882668+0000', 'last_peered': '2024-09-15T21:07:08.882668+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:08.882668+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6033, 'log_dups_size': 0, 'ondisk_log_size': 6033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49378572, 'num_objects': 6028, 'num_object_clones': 0, 'num_object_copies': 24112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6028, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6031, 'num_write_kb': 48236, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3713}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:15.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:15.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 232 20 98098 98078 3.30214 1.88281 0.0656638 0.0473022 2024-09-15T21:07:15.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:15.880 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34742/386556 objects degraded (8.988%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":301,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96639,"data_bytes":791631638,"bytes_used":2744594432,"bytes_avail":383802462208,"bytes_total":386547056640,"degraded_objects":34742,"degraded_total":386556,"degraded_ratio":0.089875723051769996},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:15.911 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:07:15.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:16.493 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:16.495 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:16.521 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-09-15T21:07:15.997112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":799659798,"num_objects":97619,"num_object_clones":0,"num_object_copies":390476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91763,"num_write_kb":733998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97676,"ondisk_log_size":97676,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2734904,"kb_used_data":1419000,"kb_used_omap":14,"kb_used_meta":1315825,"kb_avail":374752456,"statfs":{"total":386547056640,"available":383746514944,"internally_reserved":0,"allocated":1453056000,"data_stored":1452396044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1347405362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,14,9,13,14,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25268334,"num_objects":3084,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2799,"num_write_kb":-22380,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2172,"num_bytes_recovered":-17743872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004256"},"pg_stats":[{"pgid":"1.f","version":"33'6029","reported_seq":6074,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.269777+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:14.269777+0000","last_peered":"2024-09-15T21:07:14.269777+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:14.269777+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49389568,"num_objects":6029,"num_object_clones":0,"num_object_copies":24116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6029,"num_write_kb":48232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6132","reported_seq":6443,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:13.988723+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:13.988723+0000","last_peered":"2024-09-15T21:07:13.988723+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:13.988723+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6132,"log_dups_size":0,"ondisk_log_size":6132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50197772,"num_objects":6128,"num_object_clones":0,"num_object_copies":24512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6130,"num_write_kb":49036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3871}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6014","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.931373+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:11.931373+0000","last_peered":"2024-09-15T21:07:11.931373+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:11.931373+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49240710,"num_objects":6011,"num_object_clones":0,"num_object_copies":24044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6013,"num_write_kb":48094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3753}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6067","reported_seq":6136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:13.945977+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:13.945977+0000","last_peered":"2024-09-15T21:07:13.945977+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:13.945977+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6067,"log_dups_size":0,"ondisk_log_size":6067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49665292,"num_objects":6063,"num_object_clones":0,"num_object_copies":24252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6065,"num_write_kb":48509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3729}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6199","reported_seq":6254,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.092416+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:14.092416+0000","last_peered":"2024-09-15T21:07:14.092416+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:14.092416+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6199,"log_dups_size":0,"ondisk_log_size":6199,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50754828,"num_objects":6196,"num_object_clones":0,"num_object_copies":24784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":49573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6046","reported_seq":6090,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.273395+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:14.273395+0000","last_peered":"2024-09-15T21:07:14.273395+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:14.273395+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6046,"log_dups_size":0,"ondisk_log_size":6046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49528832,"num_objects":6046,"num_object_clones":0,"num_object_copies":24184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6046,"num_write_kb":48368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3730}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6256","reported_seq":6325,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:13.973045+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:13.973045+0000","last_peered":"2024-09-15T21:07:13.973045+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:13.973045+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51197196,"num_objects":6250,"num_object_clones":0,"num_object_copies":25000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3884}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6225","reported_seq":6293,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:13.994645+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:13.994645+0000","last_peered":"2024-09-15T21:07:13.994645+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:13.994645+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6225,"log_dups_size":0,"ondisk_log_size":6225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50951436,"num_objects":6220,"num_object_clones":0,"num_object_copies":24880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":49772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6075","reported_seq":6134,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.291137+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:14.291137+0000","last_peered":"2024-09-15T21:07:14.291137+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:14.291137+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49695256,"num_objects":6067,"num_object_clones":0,"num_object_copies":24268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6072,"num_write_kb":48553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3735}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6106","reported_seq":6186,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.842635+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:11.842635+0000","last_peered":"2024-09-15T21:07:11.842635+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:11.842635+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6106,"log_dups_size":0,"ondisk_log_size":6106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50001164,"num_objects":6104,"num_object_clones":0,"num_object_copies":24416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6104,"num_write_kb":48830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3769}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6025","reported_seq":10687,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:14.336950+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:14.336950+0000","last_peered":"2024-09-15T21:07:14.336950+0000","last_clean":"2024-09-15T21:07:14.336950+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:14.336950+0000","last_undegraded":"2024-09-15T21:07:14.336950+0000","last_fullsized":"2024-09-15T21:07:14.336950+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6025,"log_dups_size":0,"ondisk_log_size":6025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49291264,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":136,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6082","reported_seq":6120,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.324324+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:14.324324+0000","last_peered":"2024-09-15T21:07:14.324324+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:14.324324+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49814150,"num_objects":6081,"num_object_clones":0,"num_object_copies":24324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6081,"num_write_kb":48647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6072","reported_seq":6151,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.852246+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:11.852246+0000","last_peered":"2024-09-15T21:07:11.852246+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:11.852246+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6072,"log_dups_size":0,"ondisk_log_size":6072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49732230,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6071,"num_write_kb":48567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3785}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6075","reported_seq":6131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.131906+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:14.131906+0000","last_peered":"2024-09-15T21:07:14.131906+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:14.131906+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49730828,"num_objects":6071,"num_object_clones":0,"num_object_copies":24284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6073,"num_write_kb":48580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3792}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6140","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:11.896541+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:11.896541+0000","last_peered":"2024-09-15T21:07:11.896541+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:11.896541+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6140,"log_dups_size":0,"ondisk_log_size":6140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3856}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6133","reported_seq":6200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:14.016575+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:14.016575+0000","last_peered":"2024-09-15T21:07:14.016575+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:14.016575+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6133,"log_dups_size":0,"ondisk_log_size":6133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50197772,"num_objects":6128,"num_object_clones":0,"num_object_copies":24512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6131,"num_write_kb":49036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3813}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":799659798,"num_objects":97619,"num_object_clones":0,"num_object_copies":390476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34698,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91763,"num_write_kb":733998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":1007616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1455337472,"data_stored":1455337472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97676,"ondisk_log_size":97676,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755312,"kb_used_data":390432,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93616528,"statfs":{"total":96636764160,"available":95863324672,"internally_reserved":0,"allocated":399802368,"data_stored":399637379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752196,"kb_used_data":387828,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93619644,"statfs":{"total":96636764160,"available":95866515456,"internally_reserved":0,"allocated":397135872,"data_stored":396970883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,4,7,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051624,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":480088,"kb_used_data":249928,"kb_used_omap":4,"kb_used_meta":230139,"kb_avail":93891752,"statfs":{"total":96636764160,"available":96145154048,"internally_reserved":0,"allocated":255926272,"data_stored":255761283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":235663055},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747308,"kb_used_data":390812,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93624532,"statfs":{"total":96636764160,"available":95871520768,"internally_reserved":0,"allocated":400191488,"data_stored":400026499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,4,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":400183296,"data_stored":400183296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":256774144,"data_stored":256774144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":397881344,"data_stored":397881344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":400498688,"data_stored":400498688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:16.668 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 233 20 98334 98314 3.29588 1.84375 0.043353 0.0473941 2024-09-15T21:07:17.669 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 234 20 98602 98582 3.29074 2.09375 0.100426 0.047469 2024-09-15T21:07:18.669 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 235 20 98860 98840 3.28531 2.01562 0.131258 0.0475445 2024-09-15T21:07:19.523 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:19.669 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 236 20 99130 99110 3.28033 2.10938 0.100454 0.0476186 2024-09-15T21:07:20.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:20.112 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:20.134 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-15T21:07:19.998656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739988,"kb_used_data":1440724,"kb_used_omap":14,"kb_used_meta":1299185,"kb_avail":374747372,"statfs":{"total":386547056640,"available":383741308928,"internally_reserved":0,"allocated":1475301376,"data_stored":1474641420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1330366002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,8,0,7,10,19,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28676206,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2383,"num_write_kb":-19052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2137,"num_bytes_recovered":-17457152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004252"},"pg_stats":[{"pgid":"1.f","version":"33'6107","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.174536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:19.174536+0000","last_peered":"2024-09-15T21:07:19.174536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:19.174536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50028544,"num_objects":6107,"num_object_clones":0,"num_object_copies":24428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":48856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6196","reported_seq":6602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.051868+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:19.051868+0000","last_peered":"2024-09-15T21:07:19.051868+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:19.051868+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6196,"log_dups_size":0,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722060,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6099","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.941469+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:16.941469+0000","last_peered":"2024-09-15T21:07:16.941469+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:16.941469+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49937030,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":48774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6146","reported_seq":6213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.005824+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:19.005824+0000","last_peered":"2024-09-15T21:07:19.005824+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:19.005824+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6283","reported_seq":6338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.427604+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:19.427604+0000","last_peered":"2024-09-15T21:07:19.427604+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:19.427604+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51442956,"num_objects":6280,"num_object_clones":0,"num_object_copies":25120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6122","reported_seq":6166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.308296+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:19.308296+0000","last_peered":"2024-09-15T21:07:19.308296+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:19.308296+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50151424,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6326","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.018070+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:19.018070+0000","last_peered":"2024-09-15T21:07:19.018070+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:19.018070+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6314","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.057884+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:19.057884+0000","last_peered":"2024-09-15T21:07:19.057884+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:19.057884+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680524,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6164","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.223059+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:19.223059+0000","last_peered":"2024-09-15T21:07:19.223059+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:19.223059+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50424344,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":49265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6185","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.810580+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:16.810580+0000","last_peered":"2024-09-15T21:07:16.810580+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:16.810580+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648332,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":49462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6113","reported_seq":10775,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:19.353713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:19.353713+0000","last_peered":"2024-09-15T21:07:19.353713+0000","last_clean":"2024-09-15T21:07:19.353713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:19.353713+0000","last_undegraded":"2024-09-15T21:07:19.353713+0000","last_fullsized":"2024-09-15T21:07:19.353713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":1792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6139","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.328105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:19.328105+0000","last_peered":"2024-09-15T21:07:19.328105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:19.328105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50281094,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6134","reported_seq":6214,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.854647+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:16.854647+0000","last_peered":"2024-09-15T21:07:16.854647+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:16.854647+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240134,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6152","reported_seq":6208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.333838+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:19.333838+0000","last_peered":"2024-09-15T21:07:19.333838+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:19.333838+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6233","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.925355+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:16.925355+0000","last_peered":"2024-09-15T21:07:16.925355+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:16.925355+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6233,"log_dups_size":0,"ondisk_log_size":6233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033356,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":49845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6209","reported_seq":6276,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.026394+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:19.026394+0000","last_peered":"2024-09-15T21:07:19.026394+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:19.026394+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820364,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1476493312,"data_stored":1476493312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747956,"kb_used_data":395556,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93623884,"statfs":{"total":96636764160,"available":95870857216,"internally_reserved":0,"allocated":405049344,"data_stored":404884355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744912,"kb_used_data":393024,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93626928,"statfs":{"total":96636764160,"available":95873974272,"internally_reserved":0,"allocated":402456576,"data_stored":402291587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,4,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490588,"kb_used_data":256268,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93881252,"statfs":{"total":96636764160,"available":96134402048,"internally_reserved":0,"allocated":262418432,"data_stored":262253443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756532,"kb_used_data":395876,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615308,"statfs":{"total":96636764160,"available":95862075392,"internally_reserved":0,"allocated":405377024,"data_stored":405212035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405381120,"data_stored":405381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262172672,"data_stored":262172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403136512,"data_stored":403136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405803008,"data_stored":405803008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:20.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:20.672 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 237 20 99375 99355 3.27455 1.91406 0.0646699 0.0477055 2024-09-15T21:07:20.745 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:20.746 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:20.763 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-09-15T21:07:19.998656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739988,"kb_used_data":1440724,"kb_used_omap":14,"kb_used_meta":1299185,"kb_avail":374747372,"statfs":{"total":386547056640,"available":383741308928,"internally_reserved":0,"allocated":1475301376,"data_stored":1474641420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1330366002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,8,0,7,10,19,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28676206,"num_objects":3500,"num_object_clones":0,"num_object_copies":14000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2383,"num_write_kb":-19052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2137,"num_bytes_recovered":-17457152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004252"},"pg_stats":[{"pgid":"1.f","version":"33'6107","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.174536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:19.174536+0000","last_peered":"2024-09-15T21:07:19.174536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:19.174536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50028544,"num_objects":6107,"num_object_clones":0,"num_object_copies":24428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":48856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6196","reported_seq":6602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.051868+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:19.051868+0000","last_peered":"2024-09-15T21:07:19.051868+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:19.051868+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6196,"log_dups_size":0,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722060,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6099","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.941469+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:16.941469+0000","last_peered":"2024-09-15T21:07:16.941469+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:16.941469+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49937030,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":48774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6146","reported_seq":6213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.005824+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:19.005824+0000","last_peered":"2024-09-15T21:07:19.005824+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:19.005824+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6283","reported_seq":6338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.427604+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:19.427604+0000","last_peered":"2024-09-15T21:07:19.427604+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:19.427604+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51442956,"num_objects":6280,"num_object_clones":0,"num_object_copies":25120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6122","reported_seq":6166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.308296+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:19.308296+0000","last_peered":"2024-09-15T21:07:19.308296+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:19.308296+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50151424,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6326","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.018070+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:19.018070+0000","last_peered":"2024-09-15T21:07:19.018070+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:19.018070+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6314","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.057884+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:19.057884+0000","last_peered":"2024-09-15T21:07:19.057884+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:19.057884+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680524,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6164","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.223059+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:19.223059+0000","last_peered":"2024-09-15T21:07:19.223059+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:19.223059+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50424344,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":49265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6185","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.810580+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:16.810580+0000","last_peered":"2024-09-15T21:07:16.810580+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:16.810580+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648332,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":49462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6113","reported_seq":10775,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:19.353713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:19.353713+0000","last_peered":"2024-09-15T21:07:19.353713+0000","last_clean":"2024-09-15T21:07:19.353713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:19.353713+0000","last_undegraded":"2024-09-15T21:07:19.353713+0000","last_fullsized":"2024-09-15T21:07:19.353713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":1792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6139","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.328105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:19.328105+0000","last_peered":"2024-09-15T21:07:19.328105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:19.328105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50281094,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6134","reported_seq":6214,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.854647+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:16.854647+0000","last_peered":"2024-09-15T21:07:16.854647+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:16.854647+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240134,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6152","reported_seq":6208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.333838+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:19.333838+0000","last_peered":"2024-09-15T21:07:19.333838+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:19.333838+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6233","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.925355+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:16.925355+0000","last_peered":"2024-09-15T21:07:16.925355+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:16.925355+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6233,"log_dups_size":0,"ondisk_log_size":6233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033356,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":49845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6209","reported_seq":6276,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.026394+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:19.026394+0000","last_peered":"2024-09-15T21:07:19.026394+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:19.026394+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820364,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1476493312,"data_stored":1476493312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747956,"kb_used_data":395556,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93623884,"statfs":{"total":96636764160,"available":95870857216,"internally_reserved":0,"allocated":405049344,"data_stored":404884355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744912,"kb_used_data":393024,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93626928,"statfs":{"total":96636764160,"available":95873974272,"internally_reserved":0,"allocated":402456576,"data_stored":402291587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,4,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490588,"kb_used_data":256268,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93881252,"statfs":{"total":96636764160,"available":96134402048,"internally_reserved":0,"allocated":262418432,"data_stored":262253443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756532,"kb_used_data":395876,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615308,"statfs":{"total":96636764160,"available":95862075392,"internally_reserved":0,"allocated":405377024,"data_stored":405212035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405381120,"data_stored":405381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262172672,"data_stored":262172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403136512,"data_stored":403136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405803008,"data_stored":405803008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:20.764 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6107", 'reported_seq': 6152, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.174536+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:19.174536+0000', 'last_peered': '2024-09-15T21:07:19.174536+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:19.174536+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6107, 'log_dups_size': 0, 'ondisk_log_size': 6107, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50028544, 'num_objects': 6107, 'num_object_clones': 0, 'num_object_copies': 24428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6107, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6107, 'num_write_kb': 48856, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3831}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6196", 'reported_seq': 6602, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.051868+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:19.051868+0000', 'last_peered': '2024-09-15T21:07:19.051868+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:19.051868+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6196, 'log_dups_size': 0, 'ondisk_log_size': 6196, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50722060, 'num_objects': 6192, 'num_object_clones': 0, 'num_object_copies': 24768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6192, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6194, 'num_write_kb': 49548, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 171, 'num_bytes_recovered': 1384448, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3981}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6099", 'reported_seq': 6178, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:16.941469+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:16.941469+0000', 'last_peered': '2024-09-15T21:07:16.941469+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:16.941469+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6099, 'log_dups_size': 0, 'ondisk_log_size': 6099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49937030, 'num_objects': 6096, 'num_object_clones': 0, 'num_object_copies': 24384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6096, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6098, 'num_write_kb': 48774, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3838}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6146", 'reported_seq': 6213, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.005824+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:19.005824+0000', 'last_peered': '2024-09-15T21:07:19.005824+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:19.005824+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6146, 'log_dups_size': 0, 'ondisk_log_size': 6146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50312460, 'num_objects': 6142, 'num_object_clones': 0, 'num_object_copies': 24568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6144, 'num_write_kb': 49141, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6283", 'reported_seq': 6338, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.427604+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:19.427604+0000', 'last_peered': '2024-09-15T21:07:19.427604+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:19.427604+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6283, 'log_dups_size': 0, 'ondisk_log_size': 6283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51442956, 'num_objects': 6280, 'num_object_clones': 0, 'num_object_copies': 25120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6281, 'num_write_kb': 50245, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3969}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:20.765 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6122", 'reported_seq': 6166, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.308296+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:19.308296+0000', 'last_peered': '2024-09-15T21:07:19.308296+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:19.308296+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6122, 'log_dups_size': 0, 'ondisk_log_size': 6122, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50151424, 'num_objects': 6122, 'num_object_clones': 0, 'num_object_copies': 24488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6122, 'num_write_kb': 48976, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3806}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6326", 'reported_seq': 6396, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.018070+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:19.018070+0000', 'last_peered': '2024-09-15T21:07:19.018070+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:19.018070+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6326, 'log_dups_size': 0, 'ondisk_log_size': 6326, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51770636, 'num_objects': 6320, 'num_object_clones': 0, 'num_object_copies': 25280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6324, 'num_write_kb': 50572, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3954}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6314", 'reported_seq': 6381, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.057884+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:19.057884+0000', 'last_peered': '2024-09-15T21:07:19.057884+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:19.057884+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6314, 'log_dups_size': 0, 'ondisk_log_size': 6314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51680524, 'num_objects': 6309, 'num_object_clones': 0, 'num_object_copies': 25236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6312, 'num_write_kb': 50484, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6164", 'reported_seq': 6223, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.223059+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:19.223059+0000', 'last_peered': '2024-09-15T21:07:19.223059+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:19.223059+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6164, 'log_dups_size': 0, 'ondisk_log_size': 6164, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50424344, 'num_objects': 6156, 'num_object_clones': 0, 'num_object_copies': 24624, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6156, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6161, 'num_write_kb': 49265, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3824}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6185", 'reported_seq': 6266, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:16.810580+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:16.810580+0000', 'last_peered': '2024-09-15T21:07:16.810580+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:16.810580+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6185, 'log_dups_size': 0, 'ondisk_log_size': 6185, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50648332, 'num_objects': 6183, 'num_object_clones': 0, 'num_object_copies': 24732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6183, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6183, 'num_write_kb': 49462, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3848}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:20.766 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6139", 'reported_seq': 6178, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.328105+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:19.328105+0000', 'last_peered': '2024-09-15T21:07:19.328105+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:19.328105+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6139, 'log_dups_size': 0, 'ondisk_log_size': 6139, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50281094, 'num_objects': 6138, 'num_object_clones': 0, 'num_object_copies': 24552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6138, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6138, 'num_write_kb': 49103, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3882}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6134", 'reported_seq': 6214, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:16.854647+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:16.854647+0000', 'last_peered': '2024-09-15T21:07:16.854647+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:16.854647+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6134, 'log_dups_size': 0, 'ondisk_log_size': 6134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50240134, 'num_objects': 6133, 'num_object_clones': 0, 'num_object_copies': 24532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6133, 'num_write_kb': 49063, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3847}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6152", 'reported_seq': 6208, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.333838+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:19.333838+0000', 'last_peered': '2024-09-15T21:07:19.333838+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:19.333838+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6152, 'log_dups_size': 0, 'ondisk_log_size': 6152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50361612, 'num_objects': 6148, 'num_object_clones': 0, 'num_object_copies': 24592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6150, 'num_write_kb': 49196, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3869}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6233", 'reported_seq': 6314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:16.925355+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:16.925355+0000', 'last_peered': '2024-09-15T21:07:16.925355+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:16.925355+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6233, 'log_dups_size': 0, 'ondisk_log_size': 6233, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51033356, 'num_objects': 6230, 'num_object_clones': 0, 'num_object_copies': 24920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6231, 'num_write_kb': 49845, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:20.767 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6209", 'reported_seq': 6276, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:19.026394+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:19.026394+0000', 'last_peered': '2024-09-15T21:07:19.026394+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:19.026394+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6209, 'log_dups_size': 0, 'ondisk_log_size': 6209, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50820364, 'num_objects': 6204, 'num_object_clones': 0, 'num_object_copies': 24816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6207, 'num_write_kb': 49644, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:20.768 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:21.521 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:21.521 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34652/395460 objects degraded (8.762%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":98865,"data_bytes":809867030,"bytes_used":2805747712,"bytes_avail":383741308928,"bytes_total":386547056640,"degraded_objects":34652,"degraded_total":395460,"degraded_ratio":0.087624538512112479},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:21.557 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:07:21.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:21.669 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 238 20 99602 99582 3.26825 1.77344 0.0863704 0.0477959 2024-09-15T21:07:22.126 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:22.126 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:22.151 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-15T21:07:21.999350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739988,"kb_used_data":1440724,"kb_used_omap":14,"kb_used_meta":1299185,"kb_avail":374747372,"statfs":{"total":386547056640,"available":383741308928,"internally_reserved":0,"allocated":1475301376,"data_stored":1474641420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1330366002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,8,0,7,10,19,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20987904,"num_objects":2562,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2562,"num_write_kb":20496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004214"},"pg_stats":[{"pgid":"1.f","version":"33'6107","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.174536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:19.174536+0000","last_peered":"2024-09-15T21:07:19.174536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:19.174536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50028544,"num_objects":6107,"num_object_clones":0,"num_object_copies":24428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":48856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6196","reported_seq":6602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.051868+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:19.051868+0000","last_peered":"2024-09-15T21:07:19.051868+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:19.051868+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6196,"log_dups_size":0,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722060,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6099","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.941469+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:16.941469+0000","last_peered":"2024-09-15T21:07:16.941469+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:16.941469+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49937030,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":48774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3838}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6146","reported_seq":6213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.005824+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:19.005824+0000","last_peered":"2024-09-15T21:07:19.005824+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:19.005824+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6283","reported_seq":6338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.427604+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:19.427604+0000","last_peered":"2024-09-15T21:07:19.427604+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:19.427604+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51442956,"num_objects":6280,"num_object_clones":0,"num_object_copies":25120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6122","reported_seq":6166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.308296+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:19.308296+0000","last_peered":"2024-09-15T21:07:19.308296+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:19.308296+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50151424,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6326","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.018070+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:19.018070+0000","last_peered":"2024-09-15T21:07:19.018070+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:19.018070+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6314","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.057884+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:19.057884+0000","last_peered":"2024-09-15T21:07:19.057884+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:19.057884+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680524,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6164","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.223059+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:19.223059+0000","last_peered":"2024-09-15T21:07:19.223059+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:19.223059+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50424344,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":49265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6185","reported_seq":6266,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.810580+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:16.810580+0000","last_peered":"2024-09-15T21:07:16.810580+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:16.810580+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6185,"log_dups_size":0,"ondisk_log_size":6185,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50648332,"num_objects":6183,"num_object_clones":0,"num_object_copies":24732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":49462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3848}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6113","reported_seq":10775,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:19.353713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:19.353713+0000","last_peered":"2024-09-15T21:07:19.353713+0000","last_clean":"2024-09-15T21:07:19.353713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:19.353713+0000","last_undegraded":"2024-09-15T21:07:19.353713+0000","last_fullsized":"2024-09-15T21:07:19.353713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":1792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6139","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.328105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:19.328105+0000","last_peered":"2024-09-15T21:07:19.328105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:19.328105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50281094,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6134","reported_seq":6214,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.854647+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:16.854647+0000","last_peered":"2024-09-15T21:07:16.854647+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:16.854647+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6134,"log_dups_size":0,"ondisk_log_size":6134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50240134,"num_objects":6133,"num_object_clones":0,"num_object_copies":24532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6133,"num_write_kb":49063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3847}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6152","reported_seq":6208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.333838+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:19.333838+0000","last_peered":"2024-09-15T21:07:19.333838+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:19.333838+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6233","reported_seq":6314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:16.925355+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:16.925355+0000","last_peered":"2024-09-15T21:07:16.925355+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:16.925355+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6233,"log_dups_size":0,"ondisk_log_size":6233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51033356,"num_objects":6230,"num_object_clones":0,"num_object_copies":24920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6231,"num_write_kb":49845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3949}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6209","reported_seq":6276,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.026394+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:19.026394+0000","last_peered":"2024-09-15T21:07:19.026394+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:19.026394+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820364,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":809867030,"num_objects":98865,"num_object_clones":0,"num_object_copies":395460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93009,"num_write_kb":743966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1476493312,"data_stored":1476493312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98922,"ondisk_log_size":98922,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747956,"kb_used_data":395556,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93623884,"statfs":{"total":96636764160,"available":95870857216,"internally_reserved":0,"allocated":405049344,"data_stored":404884355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744912,"kb_used_data":393024,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93626928,"statfs":{"total":96636764160,"available":95873974272,"internally_reserved":0,"allocated":402456576,"data_stored":402291587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,1,4,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051625,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":490588,"kb_used_data":256268,"kb_used_omap":4,"kb_used_meta":234299,"kb_avail":93881252,"statfs":{"total":96636764160,"available":96134402048,"internally_reserved":0,"allocated":262418432,"data_stored":262253443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":239922895},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,1,4,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756532,"kb_used_data":395876,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615308,"statfs":{"total":96636764160,"available":95862075392,"internally_reserved":0,"allocated":405377024,"data_stored":405212035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405381120,"data_stored":405381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262172672,"data_stored":262172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":403136512,"data_stored":403136512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405803008,"data_stored":405803008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:22.672 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 239 20 99851 99831 3.26271 1.94531 0.0440866 0.0478769 2024-09-15T21:07:23.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:2024-09-15T21:07:23.667969+0000 min lat: 0.00413303 max lat: 2.73639 avg lat: 0.0479498 2024-09-15T21:07:23.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T21:07:23.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout: 240 5 100108 100103 3.25797 2.125 0.0606346 0.0479498 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Total time run: 240.055 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Total writes made: 100108 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Write size: 8192 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Object size: 8192 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Bandwidth (MB/sec): 3.25797 2024-09-15T21:07:24.670 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Stddev Bandwidth: 2.33015 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Max bandwidth (MB/sec): 13.2422 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Min bandwidth (MB/sec): 0 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Average IOPS: 417 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Stddev IOPS: 298.262 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Max IOPS: 1695 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Min IOPS: 0 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Average Latency(s): 0.0479503 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Stddev Latency(s): 0.0492991 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Max latency(s): 2.73639 2024-09-15T21:07:24.671 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Min latency(s): 0.00413303 2024-09-15T21:07:24.683 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Cleaning up (deleting benchmark objects) 2024-09-15T21:07:25.152 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:25.755 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:25.756 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:25.778 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-15T21:07:23.999946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":812578582,"num_objects":99196,"num_object_clones":0,"num_object_copies":396784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93340,"num_write_kb":746614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99253,"ondisk_log_size":99253,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759488,"kb_used_data":1451904,"kb_used_omap":14,"kb_used_meta":1307505,"kb_avail":374727872,"statfs":{"total":386547056640,"available":383721340928,"internally_reserved":0,"allocated":1486749696,"data_stored":1486089740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1338885682},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,2,7,10,25,20,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20946944,"num_objects":2557,"num_object_clones":0,"num_object_copies":10228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2557,"num_write_kb":20456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004014"},"pg_stats":[{"pgid":"1.f","version":"33'6107","reported_seq":6152,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.174536+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:19.174536+0000","last_peered":"2024-09-15T21:07:19.174536+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:19.174536+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6107,"log_dups_size":0,"ondisk_log_size":6107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50028544,"num_objects":6107,"num_object_clones":0,"num_object_copies":24428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":48856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3831}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6196","reported_seq":6602,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.051868+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:19.051868+0000","last_peered":"2024-09-15T21:07:19.051868+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:19.051868+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6196,"log_dups_size":0,"ondisk_log_size":6196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50722060,"num_objects":6192,"num_object_clones":0,"num_object_copies":24768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6194,"num_write_kb":49548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6189","reported_seq":6268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.899720+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:21.899720+0000","last_peered":"2024-09-15T21:07:21.899720+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:21.899720+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674310,"num_objects":6186,"num_object_clones":0,"num_object_copies":24744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6146","reported_seq":6213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.005824+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:19.005824+0000","last_peered":"2024-09-15T21:07:19.005824+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:19.005824+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6146,"log_dups_size":0,"ondisk_log_size":6146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50312460,"num_objects":6142,"num_object_clones":0,"num_object_copies":24568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6144,"num_write_kb":49141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3808}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6283","reported_seq":6338,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.427604+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:19.427604+0000","last_peered":"2024-09-15T21:07:19.427604+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:19.427604+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51442956,"num_objects":6280,"num_object_clones":0,"num_object_copies":25120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6281,"num_write_kb":50245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3969}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6122","reported_seq":6166,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.308296+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:19.308296+0000","last_peered":"2024-09-15T21:07:19.308296+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:19.308296+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6122,"log_dups_size":0,"ondisk_log_size":6122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50151424,"num_objects":6122,"num_object_clones":0,"num_object_copies":24488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6122,"num_write_kb":48976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3806}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6326","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.018070+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:19.018070+0000","last_peered":"2024-09-15T21:07:19.018070+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:19.018070+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51770636,"num_objects":6320,"num_object_clones":0,"num_object_copies":25280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6324,"num_write_kb":50572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3954}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6314","reported_seq":6381,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.057884+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:19.057884+0000","last_peered":"2024-09-15T21:07:19.057884+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:19.057884+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51680524,"num_objects":6309,"num_object_clones":0,"num_object_copies":25236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":50484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3835}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6164","reported_seq":6223,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.223059+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:19.223059+0000","last_peered":"2024-09-15T21:07:19.223059+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:19.223059+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6164,"log_dups_size":0,"ondisk_log_size":6164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50424344,"num_objects":6156,"num_object_clones":0,"num_object_copies":24624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6161,"num_write_kb":49265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3824}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6256","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.710251+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:21.710251+0000","last_peered":"2024-09-15T21:07:21.710251+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:21.710251+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51229964,"num_objects":6254,"num_object_clones":0,"num_object_copies":25016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6113","reported_seq":10775,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:19.353713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:19.353713+0000","last_peered":"2024-09-15T21:07:19.353713+0000","last_clean":"2024-09-15T21:07:19.353713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:19.353713+0000","last_undegraded":"2024-09-15T21:07:19.353713+0000","last_fullsized":"2024-09-15T21:07:19.353713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6113,"log_dups_size":0,"ondisk_log_size":6113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50012160,"num_objects":6105,"num_object_clones":0,"num_object_copies":24420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":224,"num_write_kb":1792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6139","reported_seq":6178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.328105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:19.328105+0000","last_peered":"2024-09-15T21:07:19.328105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:19.328105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50281094,"num_objects":6138,"num_object_clones":0,"num_object_copies":24552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6138,"num_write_kb":49103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3882}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6224","reported_seq":6301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.904670+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:21.904670+0000","last_peered":"2024-09-15T21:07:21.904670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:21.904670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50977414,"num_objects":6223,"num_object_clones":0,"num_object_copies":24892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":49783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6152","reported_seq":6208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.333838+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:19.333838+0000","last_peered":"2024-09-15T21:07:19.333838+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:19.333838+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50361612,"num_objects":6148,"num_object_clones":0,"num_object_copies":24592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":49196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3869}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6313","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.987049+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:21.987049+0000","last_peered":"2024-09-15T21:07:21.987049+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:21.987049+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51688716,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":50485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6209","reported_seq":6276,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:19.026394+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:19.026394+0000","last_peered":"2024-09-15T21:07:19.026394+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:19.026394+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50820364,"num_objects":6204,"num_object_clones":0,"num_object_copies":24816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":49644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3889}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":812578582,"num_objects":99196,"num_object_clones":0,"num_object_copies":396784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93340,"num_write_kb":746614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1487044608,"data_stored":1487044608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99253,"ondisk_log_size":99253,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747956,"kb_used_data":395556,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93623884,"statfs":{"total":96636764160,"available":95870857216,"internally_reserved":0,"allocated":405049344,"data_stored":404884355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754980,"kb_used_data":398932,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93616860,"statfs":{"total":96636764160,"available":95863664640,"internally_reserved":0,"allocated":408506368,"data_stored":408341379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500020,"kb_used_data":261540,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93871820,"statfs":{"total":96636764160,"available":96124743680,"internally_reserved":0,"allocated":267816960,"data_stored":267651971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756532,"kb_used_data":395876,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93615308,"statfs":{"total":96636764160,"available":95862075392,"internally_reserved":0,"allocated":405377024,"data_stored":405212035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,2,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405381120,"data_stored":405381120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267550720,"data_stored":267550720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408309760,"data_stored":408309760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":405803008,"data_stored":405803008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:25.779 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:26.377 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:26.377 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:26.416 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-15T21:07:26.000328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819353366,"num_objects":100023,"num_object_clones":0,"num_object_copies":400092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94167,"num_write_kb":753230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100080,"ondisk_log_size":100080,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777700,"kb_used_data":1461796,"kb_used_omap":14,"kb_used_meta":1315825,"kb_avail":374709660,"statfs":{"total":386547056640,"available":383702691840,"internally_reserved":0,"allocated":1496879104,"data_stored":1496219148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1347405362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,3,8,18,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27721728,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003897"},"pg_stats":[{"pgid":"1.f","version":"33'6176","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.507671+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:23.507671+0000","last_peered":"2024-09-15T21:07:23.507671+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:23.507671+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":6176,"num_object_clones":0,"num_object_copies":24704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6278","reported_seq":6854,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:24.078965+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:24.078965+0000","last_peered":"2024-09-15T21:07:24.078965+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:24.078965+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51393804,"num_objects":6274,"num_object_clones":0,"num_object_copies":25096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6276,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6189","reported_seq":6268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.899720+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:21.899720+0000","last_peered":"2024-09-15T21:07:21.899720+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:21.899720+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674310,"num_objects":6186,"num_object_clones":0,"num_object_copies":24744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6230","reported_seq":6299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.675566+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:23.675566+0000","last_peered":"2024-09-15T21:07:23.675566+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:23.675566+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51000588,"num_objects":6226,"num_object_clones":0,"num_object_copies":24904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6331","reported_seq":6387,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.528243+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:23.528243+0000","last_peered":"2024-09-15T21:07:23.528243+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:23.528243+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51836172,"num_objects":6328,"num_object_clones":0,"num_object_copies":25312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6188","reported_seq":6232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.648405+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:23.648405+0000","last_peered":"2024-09-15T21:07:23.648405+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:23.648405+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50692096,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6389","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.588891+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:23.588891+0000","last_peered":"2024-09-15T21:07:23.588891+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:23.588891+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52286732,"num_objects":6383,"num_object_clones":0,"num_object_copies":25532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4017}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6383","reported_seq":6452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.675598+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:23.675598+0000","last_peered":"2024-09-15T21:07:23.675598+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:23.675598+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52245772,"num_objects":6378,"num_object_clones":0,"num_object_copies":25512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6230","reported_seq":6290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.631581+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:23.631581+0000","last_peered":"2024-09-15T21:07:23.631581+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:23.631581+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50965016,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6256","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.710251+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:21.710251+0000","last_peered":"2024-09-15T21:07:21.710251+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:21.710251+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51229964,"num_objects":6254,"num_object_clones":0,"num_object_copies":25016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6184","reported_seq":10847,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:23.646012+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:23.646012+0000","last_peered":"2024-09-15T21:07:23.646012+0000","last_clean":"2024-09-15T21:07:23.646012+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:23.646012+0000","last_undegraded":"2024-09-15T21:07:23.646012+0000","last_fullsized":"2024-09-15T21:07:23.646012+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6184,"log_dups_size":0,"ondisk_log_size":6184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":6176,"num_object_clones":0,"num_object_copies":24704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":295,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6214","reported_seq":6253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.642066+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:23.642066+0000","last_peered":"2024-09-15T21:07:23.642066+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:23.642066+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50895494,"num_objects":6213,"num_object_clones":0,"num_object_copies":24852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3957}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6224","reported_seq":6301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.904670+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:21.904670+0000","last_peered":"2024-09-15T21:07:21.904670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:21.904670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50977414,"num_objects":6223,"num_object_clones":0,"num_object_copies":24892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":49783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6218","reported_seq":6274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.613110+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:23.613110+0000","last_peered":"2024-09-15T21:07:23.613110+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:23.613110+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902284,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6313","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.987049+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:21.987049+0000","last_peered":"2024-09-15T21:07:21.987049+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:21.987049+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51688716,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":50485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6277","reported_seq":6345,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.592611+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:23.592611+0000","last_peered":"2024-09-15T21:07:23.592611+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:23.592611+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6277,"log_dups_size":0,"ondisk_log_size":6277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819353366,"num_objects":100023,"num_object_clones":0,"num_object_copies":400092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94167,"num_write_kb":753230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495953408,"data_stored":1495953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100080,"ondisk_log_size":100080,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757176,"kb_used_data":400616,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93614664,"statfs":{"total":96636764160,"available":95861415936,"internally_reserved":0,"allocated":410230784,"data_stored":410065795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754980,"kb_used_data":398932,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93616860,"statfs":{"total":96636764160,"available":95863664640,"internally_reserved":0,"allocated":408506368,"data_stored":408341379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500020,"kb_used_data":261540,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93871820,"statfs":{"total":96636764160,"available":96124743680,"internally_reserved":0,"allocated":267816960,"data_stored":267651971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765524,"kb_used_data":400708,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93606316,"statfs":{"total":96636764160,"available":95852867584,"internally_reserved":0,"allocated":410324992,"data_stored":410160003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410046464,"data_stored":410046464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267550720,"data_stored":267550720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408309760,"data_stored":408309760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410046464,"data_stored":410046464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6176", 'reported_seq': 6221, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.507671+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:23.507671+0000', 'last_peered': '2024-09-15T21:07:23.507671+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:23.507671+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6176, 'log_dups_size': 0, 'ondisk_log_size': 6176, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50593792, 'num_objects': 6176, 'num_object_clones': 0, 'num_object_copies': 24704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6176, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6176, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3900}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6278", 'reported_seq': 6854, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:24.078965+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:24.078965+0000', 'last_peered': '2024-09-15T21:07:24.078965+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:24.078965+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6278, 'log_dups_size': 0, 'ondisk_log_size': 6278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51393804, 'num_objects': 6274, 'num_object_clones': 0, 'num_object_copies': 25096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2126, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6276, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 256, 'num_bytes_recovered': 2080768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4148}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:26.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6189", 'reported_seq': 6268, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:21.899720+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:21.899720+0000', 'last_peered': '2024-09-15T21:07:21.899720+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:21.899720+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6189, 'log_dups_size': 0, 'ondisk_log_size': 6189, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50674310, 'num_objects': 6186, 'num_object_clones': 0, 'num_object_copies': 24744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6188, 'num_write_kb': 49494, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3928}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6230", 'reported_seq': 6299, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.675566+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:23.675566+0000', 'last_peered': '2024-09-15T21:07:23.675566+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:23.675566+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6230, 'log_dups_size': 0, 'ondisk_log_size': 6230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51000588, 'num_objects': 6226, 'num_object_clones': 0, 'num_object_copies': 24904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6228, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6331", 'reported_seq': 6387, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.528243+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:23.528243+0000', 'last_peered': '2024-09-15T21:07:23.528243+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:23.528243+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6331, 'log_dups_size': 0, 'ondisk_log_size': 6331, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51836172, 'num_objects': 6328, 'num_object_clones': 0, 'num_object_copies': 25312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6328, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6329, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 4017}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6188", 'reported_seq': 6232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.648405+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:23.648405+0000', 'last_peered': '2024-09-15T21:07:23.648405+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:23.648405+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6188, 'log_dups_size': 0, 'ondisk_log_size': 6188, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50692096, 'num_objects': 6188, 'num_object_clones': 0, 'num_object_copies': 24752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6188, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3872}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6389", 'reported_seq': 6459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.588891+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:23.588891+0000', 'last_peered': '2024-09-15T21:07:23.588891+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:23.588891+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6389, 'log_dups_size': 0, 'ondisk_log_size': 6389, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52286732, 'num_objects': 6383, 'num_object_clones': 0, 'num_object_copies': 25532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6387, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 4017}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:26.418 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6383", 'reported_seq': 6452, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.675598+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:23.675598+0000', 'last_peered': '2024-09-15T21:07:23.675598+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:23.675598+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6383, 'log_dups_size': 0, 'ondisk_log_size': 6383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 52245772, 'num_objects': 6378, 'num_object_clones': 0, 'num_object_copies': 25512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6378, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6381, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3904}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6230", 'reported_seq': 6290, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.631581+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:23.631581+0000', 'last_peered': '2024-09-15T21:07:23.631581+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:23.631581+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6230, 'log_dups_size': 0, 'ondisk_log_size': 6230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50965016, 'num_objects': 6222, 'num_object_clones': 0, 'num_object_copies': 24888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6222, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6227, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3890}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6256", 'reported_seq': 6337, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:21.710251+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:21.710251+0000', 'last_peered': '2024-09-15T21:07:21.710251+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:21.710251+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6256, 'log_dups_size': 0, 'ondisk_log_size': 6256, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51229964, 'num_objects': 6254, 'num_object_clones': 0, 'num_object_copies': 25016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6254, 'num_write_kb': 50030, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3919}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6214", 'reported_seq': 6253, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.642066+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:23.642066+0000', 'last_peered': '2024-09-15T21:07:23.642066+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:23.642066+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6214, 'log_dups_size': 0, 'ondisk_log_size': 6214, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50895494, 'num_objects': 6213, 'num_object_clones': 0, 'num_object_copies': 24852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6213, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6224", 'reported_seq': 6301, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:21.904670+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:21.904670+0000', 'last_peered': '2024-09-15T21:07:21.904670+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:21.904670+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6224, 'log_dups_size': 0, 'ondisk_log_size': 6224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50977414, 'num_objects': 6223, 'num_object_clones': 0, 'num_object_copies': 24892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6223, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6223, 'num_write_kb': 49783, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:26.419 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6218", 'reported_seq': 6274, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.613110+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:23.613110+0000', 'last_peered': '2024-09-15T21:07:23.613110+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:23.613110+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6218, 'log_dups_size': 0, 'ondisk_log_size': 6218, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50902284, 'num_objects': 6214, 'num_object_clones': 0, 'num_object_copies': 24856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6216, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3935}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:26.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:26.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6313", 'reported_seq': 6396, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:21.987049+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:21.987049+0000', 'last_peered': '2024-09-15T21:07:21.987049+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:21.987049+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6313, 'log_dups_size': 0, 'ondisk_log_size': 6313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51688716, 'num_objects': 6310, 'num_object_clones': 0, 'num_object_copies': 25240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6311, 'num_write_kb': 50485, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4029}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:26.420 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:26.420 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6277", 'reported_seq': 6345, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:23.592611+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:23.592611+0000', 'last_peered': '2024-09-15T21:07:23.592611+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:23.592611+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6277, 'log_dups_size': 0, 'ondisk_log_size': 6277, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51377420, 'num_objects': 6272, 'num_object_clones': 0, 'num_object_copies': 25088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6275, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3957}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:26.421 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:27.248 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:27.248 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34567/400092 objects degraded (8.640%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":313,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":100023,"data_bytes":819353366,"bytes_used":2844364800,"bytes_avail":383702691840,"bytes_total":386547056640,"degraded_objects":34567,"degraded_total":400092,"degraded_ratio":0.086397628545434563,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":119427,"recovering_keys_per_sec":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"write_bytes_sec":2309393,"read_op_per_sec":0,"write_op_per_sec":281},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:27.273 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:27.273 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:27.862 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:27.863 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:27.890 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-15T21:07:26.000328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819353366,"num_objects":100023,"num_object_clones":0,"num_object_copies":400092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94167,"num_write_kb":753230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100080,"ondisk_log_size":100080,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777700,"kb_used_data":1461796,"kb_used_omap":14,"kb_used_meta":1315825,"kb_avail":374709660,"statfs":{"total":386547056640,"available":383702691840,"internally_reserved":0,"allocated":1496879104,"data_stored":1496219148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1347405362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,8,3,8,18,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27721728,"num_objects":3384,"num_object_clones":0,"num_object_copies":13536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-175,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3384,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":1433600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003897"},"pg_stats":[{"pgid":"1.f","version":"33'6176","reported_seq":6221,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.507671+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:23.507671+0000","last_peered":"2024-09-15T21:07:23.507671+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:23.507671+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6176,"log_dups_size":0,"ondisk_log_size":6176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":6176,"num_object_clones":0,"num_object_copies":24704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6278","reported_seq":6854,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:24.078965+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:24.078965+0000","last_peered":"2024-09-15T21:07:24.078965+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:24.078965+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6278,"log_dups_size":0,"ondisk_log_size":6278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51393804,"num_objects":6274,"num_object_clones":0,"num_object_copies":25096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2126,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6276,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4148}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6189","reported_seq":6268,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.899720+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:21.899720+0000","last_peered":"2024-09-15T21:07:21.899720+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:21.899720+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50674310,"num_objects":6186,"num_object_clones":0,"num_object_copies":24744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3928}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6230","reported_seq":6299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.675566+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:23.675566+0000","last_peered":"2024-09-15T21:07:23.675566+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:23.675566+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51000588,"num_objects":6226,"num_object_clones":0,"num_object_copies":24904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6331","reported_seq":6387,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.528243+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:23.528243+0000","last_peered":"2024-09-15T21:07:23.528243+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:23.528243+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6331,"log_dups_size":0,"ondisk_log_size":6331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51836172,"num_objects":6328,"num_object_clones":0,"num_object_copies":25312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6329,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":4017}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6188","reported_seq":6232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.648405+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:23.648405+0000","last_peered":"2024-09-15T21:07:23.648405+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:23.648405+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50692096,"num_objects":6188,"num_object_clones":0,"num_object_copies":24752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3872}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6389","reported_seq":6459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.588891+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:23.588891+0000","last_peered":"2024-09-15T21:07:23.588891+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:23.588891+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52286732,"num_objects":6383,"num_object_clones":0,"num_object_copies":25532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6387,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":4017}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6383","reported_seq":6452,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.675598+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:23.675598+0000","last_peered":"2024-09-15T21:07:23.675598+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:23.675598+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52245772,"num_objects":6378,"num_object_clones":0,"num_object_copies":25512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3904}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6230","reported_seq":6290,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.631581+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:23.631581+0000","last_peered":"2024-09-15T21:07:23.631581+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:23.631581+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6230,"log_dups_size":0,"ondisk_log_size":6230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50965016,"num_objects":6222,"num_object_clones":0,"num_object_copies":24888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6227,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6256","reported_seq":6337,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.710251+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:21.710251+0000","last_peered":"2024-09-15T21:07:21.710251+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:21.710251+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51229964,"num_objects":6254,"num_object_clones":0,"num_object_copies":25016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6254,"num_write_kb":50030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6184","reported_seq":10847,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:23.646012+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:23.646012+0000","last_peered":"2024-09-15T21:07:23.646012+0000","last_clean":"2024-09-15T21:07:23.646012+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:23.646012+0000","last_undegraded":"2024-09-15T21:07:23.646012+0000","last_fullsized":"2024-09-15T21:07:23.646012+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6184,"log_dups_size":0,"ondisk_log_size":6184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":6176,"num_object_clones":0,"num_object_copies":24704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":295,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6214","reported_seq":6253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.642066+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:23.642066+0000","last_peered":"2024-09-15T21:07:23.642066+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:23.642066+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6214,"log_dups_size":0,"ondisk_log_size":6214,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50895494,"num_objects":6213,"num_object_clones":0,"num_object_copies":24852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6213,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3957}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6224","reported_seq":6301,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.904670+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:21.904670+0000","last_peered":"2024-09-15T21:07:21.904670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:21.904670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6224,"log_dups_size":0,"ondisk_log_size":6224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50977414,"num_objects":6223,"num_object_clones":0,"num_object_copies":24892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6223,"num_write_kb":49783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3937}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6218","reported_seq":6274,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.613110+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:23.613110+0000","last_peered":"2024-09-15T21:07:23.613110+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:23.613110+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50902284,"num_objects":6214,"num_object_clones":0,"num_object_copies":24856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3935}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6313","reported_seq":6396,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:21.987049+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:21.987049+0000","last_peered":"2024-09-15T21:07:21.987049+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:21.987049+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51688716,"num_objects":6310,"num_object_clones":0,"num_object_copies":25240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6311,"num_write_kb":50485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4029}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6277","reported_seq":6345,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:23.592611+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:23.592611+0000","last_peered":"2024-09-15T21:07:23.592611+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:23.592611+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6277,"log_dups_size":0,"ondisk_log_size":6277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51377420,"num_objects":6272,"num_object_clones":0,"num_object_copies":25088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6275,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3957}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":819353366,"num_objects":100023,"num_object_clones":0,"num_object_copies":400092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94167,"num_write_kb":753230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":2080768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1495953408,"data_stored":1495953408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100080,"ondisk_log_size":100080,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757176,"kb_used_data":400616,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93614664,"statfs":{"total":96636764160,"available":95861415936,"internally_reserved":0,"allocated":410230784,"data_stored":410065795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,3,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754980,"kb_used_data":398932,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93616860,"statfs":{"total":96636764160,"available":95863664640,"internally_reserved":0,"allocated":408506368,"data_stored":408341379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051626,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":500020,"kb_used_data":261540,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93871820,"statfs":{"total":96636764160,"available":96124743680,"internally_reserved":0,"allocated":267816960,"data_stored":267651971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,6,9,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765524,"kb_used_data":400708,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93606316,"statfs":{"total":96636764160,"available":95852867584,"internally_reserved":0,"allocated":410324992,"data_stored":410160003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,0,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":410046464,"data_stored":410046464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267550720,"data_stored":267550720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":408309760,"data_stored":408309760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":410046464,"data_stored":410046464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:30.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:31.500 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:31.501 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:31.535 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-15T21:07:30.001834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765080,"kb_used_data":1453316,"kb_used_omap":14,"kb_used_meta":1311665,"kb_avail":374722280,"statfs":{"total":386547056640,"available":383715614720,"internally_reserved":0,"allocated":1488195584,"data_stored":1487535628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1343145522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,11,13,15,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9084952,"num_objects":1110,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3282,"num_write_kb":17561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":1974272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003814"},"pg_stats":[{"pgid":"1.f","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.481148+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:29.481148+0000","last_peered":"2024-09-15T21:07:29.481148+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:29.481148+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49905664,"num_objects":6092,"num_object_clones":0,"num_object_copies":24368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":7145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.054028+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:29.054028+0000","last_peered":"2024-09-15T21:07:29.054028+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:29.054028+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50812172,"num_objects":6203,"num_object_clones":0,"num_object_copies":24812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6262","reported_seq":6342,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.006782+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:27.006782+0000","last_peered":"2024-09-15T21:07:27.006782+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:27.006782+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50543262,"num_objects":6171,"num_object_clones":0,"num_object_copies":24684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6261,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6297","reported_seq":6366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.998468+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:28.998468+0000","last_peered":"2024-09-15T21:07:28.998468+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:28.998468+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451724,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.142864+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:29.142864+0000","last_peered":"2024-09-15T21:07:29.142864+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:29.142864+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6267","reported_seq":6307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.376545+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:29.376545+0000","last_peered":"2024-09-15T21:07:29.376545+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:29.376545+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50044928,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6470","reported_seq":6539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.019741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:29.019741+0000","last_peered":"2024-09-15T21:07:29.019741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:29.019741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623180,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3936}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6462","reported_seq":6531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.071266+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:29.071266+0000","last_peered":"2024-09-15T21:07:29.071266+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:29.071266+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598604,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6315","reported_seq":6375,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.250399+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:29.250399+0000","last_peered":"2024-09-15T21:07:29.250399+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:29.250399+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50268696,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6311","reported_seq":6392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.033474+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:27.033474+0000","last_peered":"2024-09-15T21:07:27.033474+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:27.033474+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6264","reported_seq":10927,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:29.458025+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:29.458025+0000","last_peered":"2024-09-15T21:07:29.458025+0000","last_clean":"2024-09-15T21:07:29.458025+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:29.458025+0000","last_undegraded":"2024-09-15T21:07:29.458025+0000","last_fullsized":"2024-09-15T21:07:29.458025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6301","reported_seq":6340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.467974+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:29.467974+0000","last_peered":"2024-09-15T21:07:29.467974+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:29.467974+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50182790,"num_objects":6126,"num_object_clones":0,"num_object_copies":24504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6300,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6283","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:26.757356+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:26.757356+0000","last_peered":"2024-09-15T21:07:26.757356+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:26.757356+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50936454,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6291","reported_seq":6347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.476297+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:29.476297+0000","last_peered":"2024-09-15T21:07:29.476297+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:29.476297+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50304268,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6383","reported_seq":6465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.062159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:27.062159+0000","last_peered":"2024-09-15T21:07:27.062159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:27.062159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51590412,"num_objects":6298,"num_object_clones":0,"num_object_copies":25192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6354","reported_seq":6422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.994591+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:28.994591+0000","last_peered":"2024-09-15T21:07:28.994591+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:28.994591+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746636,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486430208,"data_stored":1486430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756404,"kb_used_data":395664,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93615436,"statfs":{"total":96636764160,"available":95862206464,"internally_reserved":0,"allocated":405159936,"data_stored":404994947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758552,"kb_used_data":398344,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613288,"statfs":{"total":96636764160,"available":95860006912,"internally_reserved":0,"allocated":407904256,"data_stored":407739267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501220,"kb_used_data":262740,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93870620,"statfs":{"total":96636764160,"available":96123514880,"internally_reserved":0,"allocated":269045760,"data_stored":268880771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748904,"kb_used_data":396568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93622936,"statfs":{"total":96636764160,"available":95869886464,"internally_reserved":0,"allocated":406085632,"data_stored":405920643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405327872,"data_stored":405327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268664832,"data_stored":268664832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407568384,"data_stored":407568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":404869120,"data_stored":404869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:31.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:32.111 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:32.112 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:32.135 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-15T21:07:32.002696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765080,"kb_used_data":1453316,"kb_used_omap":14,"kb_used_meta":1311665,"kb_avail":374722280,"statfs":{"total":386547056640,"available":383715614720,"internally_reserved":0,"allocated":1488195584,"data_stored":1487535628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1343145522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,11,13,15,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1490968,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2355,"num_write_kb":10145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004040"},"pg_stats":[{"pgid":"1.f","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.481148+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:29.481148+0000","last_peered":"2024-09-15T21:07:29.481148+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:29.481148+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49905664,"num_objects":6092,"num_object_clones":0,"num_object_copies":24368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":7145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.054028+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:29.054028+0000","last_peered":"2024-09-15T21:07:29.054028+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:29.054028+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50812172,"num_objects":6203,"num_object_clones":0,"num_object_copies":24812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6262","reported_seq":6342,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.006782+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:27.006782+0000","last_peered":"2024-09-15T21:07:27.006782+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:27.006782+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50543262,"num_objects":6171,"num_object_clones":0,"num_object_copies":24684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6261,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6297","reported_seq":6366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.998468+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:28.998468+0000","last_peered":"2024-09-15T21:07:28.998468+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:28.998468+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451724,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.142864+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:29.142864+0000","last_peered":"2024-09-15T21:07:29.142864+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:29.142864+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6267","reported_seq":6307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.376545+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:29.376545+0000","last_peered":"2024-09-15T21:07:29.376545+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:29.376545+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50044928,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6470","reported_seq":6539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.019741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:29.019741+0000","last_peered":"2024-09-15T21:07:29.019741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:29.019741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623180,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3936}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6462","reported_seq":6531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.071266+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:29.071266+0000","last_peered":"2024-09-15T21:07:29.071266+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:29.071266+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598604,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6315","reported_seq":6375,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.250399+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:29.250399+0000","last_peered":"2024-09-15T21:07:29.250399+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:29.250399+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50268696,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6311","reported_seq":6392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.033474+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:27.033474+0000","last_peered":"2024-09-15T21:07:27.033474+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:27.033474+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6264","reported_seq":10927,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:29.458025+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:29.458025+0000","last_peered":"2024-09-15T21:07:29.458025+0000","last_clean":"2024-09-15T21:07:29.458025+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:29.458025+0000","last_undegraded":"2024-09-15T21:07:29.458025+0000","last_fullsized":"2024-09-15T21:07:29.458025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6301","reported_seq":6340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.467974+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:29.467974+0000","last_peered":"2024-09-15T21:07:29.467974+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:29.467974+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50182790,"num_objects":6126,"num_object_clones":0,"num_object_copies":24504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6300,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6283","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:26.757356+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:26.757356+0000","last_peered":"2024-09-15T21:07:26.757356+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:26.757356+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50936454,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6291","reported_seq":6347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.476297+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:29.476297+0000","last_peered":"2024-09-15T21:07:29.476297+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:29.476297+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50304268,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6383","reported_seq":6465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.062159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:27.062159+0000","last_peered":"2024-09-15T21:07:27.062159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:27.062159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51590412,"num_objects":6298,"num_object_clones":0,"num_object_copies":25192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6354","reported_seq":6422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.994591+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:28.994591+0000","last_peered":"2024-09-15T21:07:28.994591+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:28.994591+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746636,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486430208,"data_stored":1486430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756404,"kb_used_data":395664,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93615436,"statfs":{"total":96636764160,"available":95862206464,"internally_reserved":0,"allocated":405159936,"data_stored":404994947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758552,"kb_used_data":398344,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613288,"statfs":{"total":96636764160,"available":95860006912,"internally_reserved":0,"allocated":407904256,"data_stored":407739267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501220,"kb_used_data":262740,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93870620,"statfs":{"total":96636764160,"available":96123514880,"internally_reserved":0,"allocated":269045760,"data_stored":268880771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748904,"kb_used_data":396568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93622936,"statfs":{"total":96636764160,"available":95869886464,"internally_reserved":0,"allocated":406085632,"data_stored":405920643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405327872,"data_stored":405327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268664832,"data_stored":268664832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407568384,"data_stored":407568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":404869120,"data_stored":404869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:32.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:32.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6260", 'reported_seq': 6304, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.481148+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:29.481148+0000', 'last_peered': '2024-09-15T21:07:29.481148+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:29.481148+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6260, 'log_dups_size': 0, 'ondisk_log_size': 6260, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49905664, 'num_objects': 6092, 'num_object_clones': 0, 'num_object_copies': 24368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6092, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6260, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3816}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:32.136 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6349", 'reported_seq': 7145, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.054028+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:29.054028+0000', 'last_peered': '2024-09-15T21:07:29.054028+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:29.054028+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6349, 'log_dups_size': 0, 'ondisk_log_size': 6349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50812172, 'num_objects': 6203, 'num_object_clones': 0, 'num_object_copies': 24812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2016, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6347, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 366, 'num_bytes_recovered': 2981888, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4187}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:32.136 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6262", 'reported_seq': 6342, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:27.006782+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:27.006782+0000', 'last_peered': '2024-09-15T21:07:27.006782+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:27.006782+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6262, 'log_dups_size': 0, 'ondisk_log_size': 6262, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50543262, 'num_objects': 6171, 'num_object_clones': 0, 'num_object_copies': 24684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6171, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6261, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3913}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6297", 'reported_seq': 6366, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:28.998468+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:28.998468+0000', 'last_peered': '2024-09-15T21:07:28.998468+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:28.998468+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6297, 'log_dups_size': 0, 'ondisk_log_size': 6297, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50451724, 'num_objects': 6159, 'num_object_clones': 0, 'num_object_copies': 24636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6159, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6295, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6408", 'reported_seq': 6464, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.142864+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:29.142864+0000', 'last_peered': '2024-09-15T21:07:29.142864+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:29.142864+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6408, 'log_dups_size': 0, 'ondisk_log_size': 6408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51205388, 'num_objects': 6251, 'num_object_clones': 0, 'num_object_copies': 25004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6406, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3940}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.137 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6267", 'reported_seq': 6307, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.376545+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:29.376545+0000', 'last_peered': '2024-09-15T21:07:29.376545+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:29.376545+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6267, 'log_dups_size': 0, 'ondisk_log_size': 6267, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50044928, 'num_objects': 6109, 'num_object_clones': 0, 'num_object_copies': 24436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6267, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3793}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6470", 'reported_seq': 6539, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.019741+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:29.019741+0000', 'last_peered': '2024-09-15T21:07:29.019741+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:29.019741+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6470, 'log_dups_size': 0, 'ondisk_log_size': 6470, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51623180, 'num_objects': 6302, 'num_object_clones': 0, 'num_object_copies': 25208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6468, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3936}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6462", 'reported_seq': 6531, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.071266+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:29.071266+0000', 'last_peered': '2024-09-15T21:07:29.071266+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:29.071266+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6462, 'log_dups_size': 0, 'ondisk_log_size': 6462, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51598604, 'num_objects': 6299, 'num_object_clones': 0, 'num_object_copies': 25196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6299, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6460, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6315", 'reported_seq': 6375, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.250399+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:29.250399+0000', 'last_peered': '2024-09-15T21:07:29.250399+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:29.250399+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6315, 'log_dups_size': 0, 'ondisk_log_size': 6315, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50268696, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 24548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6312, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3805}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:32.138 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6311", 'reported_seq': 6392, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:27.033474+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:27.033474+0000', 'last_peered': '2024-09-15T21:07:27.033474+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:27.033474+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6311, 'log_dups_size': 0, 'ondisk_log_size': 6311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51205388, 'num_objects': 6251, 'num_object_clones': 0, 'num_object_copies': 25004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6309, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3916}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6301", 'reported_seq': 6340, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.467974+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:29.467974+0000', 'last_peered': '2024-09-15T21:07:29.467974+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:29.467974+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6301, 'log_dups_size': 0, 'ondisk_log_size': 6301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50182790, 'num_objects': 6126, 'num_object_clones': 0, 'num_object_copies': 24504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6300, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3870}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6283", 'reported_seq': 6363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:26.757356+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:26.757356+0000', 'last_peered': '2024-09-15T21:07:26.757356+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:26.757356+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6283, 'log_dups_size': 0, 'ondisk_log_size': 6283, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50936454, 'num_objects': 6218, 'num_object_clones': 0, 'num_object_copies': 24872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6218, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6282, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3932}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6291", 'reported_seq': 6347, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:29.476297+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:29.476297+0000', 'last_peered': '2024-09-15T21:07:29.476297+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:29.476297+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6291, 'log_dups_size': 0, 'ondisk_log_size': 6291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50304268, 'num_objects': 6141, 'num_object_clones': 0, 'num_object_copies': 24564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6289, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3862}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6383", 'reported_seq': 6465, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:27.062159+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:27.062159+0000', 'last_peered': '2024-09-15T21:07:27.062159+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:27.062159+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6383, 'log_dups_size': 0, 'ondisk_log_size': 6383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51590412, 'num_objects': 6298, 'num_object_clones': 0, 'num_object_copies': 25192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6381, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 4017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:32.139 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:32.140 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6354", 'reported_seq': 6422, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:28.994591+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:28.994591+0000', 'last_peered': '2024-09-15T21:07:28.994591+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:28.994591+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6354, 'log_dups_size': 0, 'ondisk_log_size': 6354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50746636, 'num_objects': 6195, 'num_object_clones': 0, 'num_object_copies': 24780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6352, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:32.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:32.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:32.872 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34457/396192 objects degraded (8.697%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":318,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":99048,"data_bytes":811357998,"bytes_used":2831441920,"bytes_avail":383715614720,"bytes_total":386547056640,"degraded_objects":34457,"degraded_total":396192,"degraded_ratio":0.086970458767466285,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":133075,"recovering_keys_per_sec":0,"num_objects_recovered":195,"num_bytes_recovered":1597440,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":865415,"read_op_per_sec":0,"write_op_per_sec":196},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:32.893 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:32.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:33.451 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:33.452 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:33.477 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-15T21:07:32.002696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765080,"kb_used_data":1453316,"kb_used_omap":14,"kb_used_meta":1311665,"kb_avail":374722280,"statfs":{"total":386547056640,"available":383715614720,"internally_reserved":0,"allocated":1488195584,"data_stored":1487535628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1343145522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,11,13,15,9,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1490968,"num_objects":183,"num_object_clones":0,"num_object_copies":732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2355,"num_write_kb":10145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":195,"num_bytes_recovered":1597440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004040"},"pg_stats":[{"pgid":"1.f","version":"33'6260","reported_seq":6304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.481148+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:29.481148+0000","last_peered":"2024-09-15T21:07:29.481148+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:29.481148+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6260,"log_dups_size":0,"ondisk_log_size":6260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49905664,"num_objects":6092,"num_object_clones":0,"num_object_copies":24368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6260,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3816}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6349","reported_seq":7145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.054028+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:29.054028+0000","last_peered":"2024-09-15T21:07:29.054028+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:29.054028+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50812172,"num_objects":6203,"num_object_clones":0,"num_object_copies":24812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2016,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6347,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":366,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4187}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6262","reported_seq":6342,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.006782+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:27.006782+0000","last_peered":"2024-09-15T21:07:27.006782+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:27.006782+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50543262,"num_objects":6171,"num_object_clones":0,"num_object_copies":24684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6171,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6261,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6297","reported_seq":6366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.998468+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:28.998468+0000","last_peered":"2024-09-15T21:07:28.998468+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:28.998468+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50451724,"num_objects":6159,"num_object_clones":0,"num_object_copies":24636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6408","reported_seq":6464,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.142864+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:29.142864+0000","last_peered":"2024-09-15T21:07:29.142864+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:29.142864+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6267","reported_seq":6307,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.376545+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:29.376545+0000","last_peered":"2024-09-15T21:07:29.376545+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:29.376545+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50044928,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6267,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6470","reported_seq":6539,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.019741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:29.019741+0000","last_peered":"2024-09-15T21:07:29.019741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:29.019741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51623180,"num_objects":6302,"num_object_clones":0,"num_object_copies":25208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6468,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3936}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6462","reported_seq":6531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.071266+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:29.071266+0000","last_peered":"2024-09-15T21:07:29.071266+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:29.071266+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51598604,"num_objects":6299,"num_object_clones":0,"num_object_copies":25196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6460,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6315","reported_seq":6375,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.250399+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:29.250399+0000","last_peered":"2024-09-15T21:07:29.250399+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:29.250399+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6315,"log_dups_size":0,"ondisk_log_size":6315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50268696,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6311","reported_seq":6392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.033474+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:27.033474+0000","last_peered":"2024-09-15T21:07:27.033474+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:27.033474+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6311,"log_dups_size":0,"ondisk_log_size":6311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51205388,"num_objects":6251,"num_object_clones":0,"num_object_copies":25004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6309,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3916}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6264","reported_seq":10927,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:29.458025+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:29.458025+0000","last_peered":"2024-09-15T21:07:29.458025+0000","last_clean":"2024-09-15T21:07:29.458025+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:29.458025+0000","last_undegraded":"2024-09-15T21:07:29.458025+0000","last_fullsized":"2024-09-15T21:07:29.458025+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6264,"log_dups_size":0,"ondisk_log_size":6264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":6096,"num_object_clones":0,"num_object_copies":24384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6301","reported_seq":6340,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.467974+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:29.467974+0000","last_peered":"2024-09-15T21:07:29.467974+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:29.467974+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6301,"log_dups_size":0,"ondisk_log_size":6301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50182790,"num_objects":6126,"num_object_clones":0,"num_object_copies":24504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6300,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3870}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6283","reported_seq":6363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:26.757356+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:26.757356+0000","last_peered":"2024-09-15T21:07:26.757356+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:26.757356+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50936454,"num_objects":6218,"num_object_clones":0,"num_object_copies":24872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3932}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6291","reported_seq":6347,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:29.476297+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:29.476297+0000","last_peered":"2024-09-15T21:07:29.476297+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:29.476297+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6291,"log_dups_size":0,"ondisk_log_size":6291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50304268,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6289,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3862}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6383","reported_seq":6465,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:27.062159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:27.062159+0000","last_peered":"2024-09-15T21:07:27.062159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:27.062159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51590412,"num_objects":6298,"num_object_clones":0,"num_object_copies":25192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6381,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":4017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6354","reported_seq":6422,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:28.994591+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:28.994591+0000","last_peered":"2024-09-15T21:07:28.994591+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:28.994591+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6354,"log_dups_size":0,"ondisk_log_size":6354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50746636,"num_objects":6195,"num_object_clones":0,"num_object_copies":24780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":811357998,"num_objects":99048,"num_object_clones":0,"num_object_copies":396192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34457,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99048,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95364,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":2981888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1486430208,"data_stored":1486430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101277,"ondisk_log_size":101277,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":756404,"kb_used_data":395664,"kb_used_omap":4,"kb_used_meta":360699,"kb_avail":93615436,"statfs":{"total":96636764160,"available":95862206464,"internally_reserved":0,"allocated":405159936,"data_stored":404994947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":369356496},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,3,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758552,"kb_used_data":398344,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93613288,"statfs":{"total":96636764160,"available":95860006912,"internally_reserved":0,"allocated":407904256,"data_stored":407739267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051627,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":501220,"kb_used_data":262740,"kb_used_omap":4,"kb_used_meta":238459,"kb_avail":93870620,"statfs":{"total":96636764160,"available":96123514880,"internally_reserved":0,"allocated":269045760,"data_stored":268880771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":244182735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":748904,"kb_used_data":396568,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93622936,"statfs":{"total":96636764160,"available":95869886464,"internally_reserved":0,"allocated":406085632,"data_stored":405920643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":405327872,"data_stored":405327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268664832,"data_stored":268664832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":407568384,"data_stored":407568384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":404869120,"data_stored":404869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:36.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:37.068 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:37.069 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:37.088 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T21:07:36.004117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":800667438,"num_objects":97743,"num_object_clones":0,"num_object_copies":390972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96669,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102582,"ondisk_log_size":102582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771204,"kb_used_data":1437236,"kb_used_omap":14,"kb_used_meta":1333873,"kb_avail":374716156,"statfs":{"total":386547056640,"available":383709343744,"internally_reserved":0,"allocated":1471729664,"data_stored":1471069708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1365886514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,14,11,24,10,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11911144,"num_objects":-1453,"num_object_clones":0,"num_object_copies":-5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3329,"num_write_kb":7497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004170"},"pg_stats":[{"pgid":"1.f","version":"33'6343","reported_seq":6388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.478705+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:34.478705+0000","last_peered":"2024-09-15T21:07:34.478705+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:34.478705+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49225728,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6436","reported_seq":7314,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.074458+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:34.074458+0000","last_peered":"2024-09-15T21:07:34.074458+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:34.074458+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6340","reported_seq":6420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.921795+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:31.921795+0000","last_peered":"2024-09-15T21:07:31.921795+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:31.921795+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49904286,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6339,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6379","reported_seq":6447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.111464+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:34.111464+0000","last_peered":"2024-09-15T21:07:34.111464+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:34.111464+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6491","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.522308+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:34.522308+0000","last_peered":"2024-09-15T21:07:34.522308+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:34.522308+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6356","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.472618+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:34.472618+0000","last_peered":"2024-09-15T21:07:34.472618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:34.472618+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6356,"log_dups_size":0,"ondisk_log_size":6356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49315840,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6543","reported_seq":6610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.077387+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:34.077387+0000","last_peered":"2024-09-15T21:07:34.077387+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:34.077387+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025164,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3863}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6561","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.030664+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:34.030664+0000","last_peered":"2024-09-15T21:07:34.030664+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:34.030664+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787596,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6409","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.404408+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:34.404408+0000","last_peered":"2024-09-15T21:07:34.404408+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:34.404408+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49498648,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6390","reported_seq":6471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.923574+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:31.923574+0000","last_peered":"2024-09-15T21:07:31.923574+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:31.923574+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50558220,"num_objects":6172,"num_object_clones":0,"num_object_copies":24688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6343","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:34.406657+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:34.406657+0000","last_peered":"2024-09-15T21:07:34.406657+0000","last_clean":"2024-09-15T21:07:34.406657+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:34.406657+0000","last_undegraded":"2024-09-15T21:07:34.406657+0000","last_fullsized":"2024-09-15T21:07:34.406657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49291264,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6366","reported_seq":6405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.299640+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:34.299640+0000","last_peered":"2024-09-15T21:07:34.299640+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:34.299640+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49650310,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6360","reported_seq":6440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.953879+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:31.953879+0000","last_peered":"2024-09-15T21:07:31.953879+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:31.953879+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6360,"log_dups_size":0,"ondisk_log_size":6360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50305670,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3855}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6370","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.395213+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:34.395213+0000","last_peered":"2024-09-15T21:07:34.395213+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:34.395213+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6370,"log_dups_size":0,"ondisk_log_size":6370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6455","reported_seq":6538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.947445+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:31.947445+0000","last_peered":"2024-09-15T21:07:31.947445+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:31.947445+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51000588,"num_objects":6226,"num_object_clones":0,"num_object_copies":24904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3945}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6440","reported_seq":6508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.138359+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:34.138359+0000","last_peered":"2024-09-15T21:07:34.138359+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:34.138359+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50042124,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6438,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":800667438,"num_objects":97743,"num_object_clones":0,"num_object_copies":390972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96669,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1469759488,"data_stored":1469759488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102582,"ondisk_log_size":102582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755464,"kb_used_data":390568,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93616376,"statfs":{"total":96636764160,"available":95863169024,"internally_reserved":0,"allocated":399941632,"data_stored":399776643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,7,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757420,"kb_used_data":393052,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614420,"statfs":{"total":96636764160,"available":95861166080,"internally_reserved":0,"allocated":402485248,"data_stored":402320259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510404,"kb_used_data":262196,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93861436,"statfs":{"total":96636764160,"available":96114110464,"internally_reserved":0,"allocated":268488704,"data_stored":268323715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747916,"kb_used_data":391420,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93623924,"statfs":{"total":96636764160,"available":95870898176,"internally_reserved":0,"allocated":400814080,"data_stored":400649091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,2,9,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399994880,"data_stored":399994880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268128256,"data_stored":268128256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402092032,"data_stored":402092032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":399544320,"data_stored":399544320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:37.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:37.680 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:37.681 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:37.707 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-15T21:07:36.004117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":800667438,"num_objects":97743,"num_object_clones":0,"num_object_copies":390972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96669,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102582,"ondisk_log_size":102582,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2771204,"kb_used_data":1437236,"kb_used_omap":14,"kb_used_meta":1333873,"kb_avail":374716156,"statfs":{"total":386547056640,"available":383709343744,"internally_reserved":0,"allocated":1471729664,"data_stored":1471069708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1365886514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,14,11,24,10,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11911144,"num_objects":-1453,"num_object_clones":0,"num_object_copies":-5812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1453,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3329,"num_write_kb":7497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":1933312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004170"},"pg_stats":[{"pgid":"1.f","version":"33'6343","reported_seq":6388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.478705+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:34.478705+0000","last_peered":"2024-09-15T21:07:34.478705+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:34.478705+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49225728,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6436","reported_seq":7314,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.074458+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:34.074458+0000","last_peered":"2024-09-15T21:07:34.074458+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:34.074458+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6340","reported_seq":6420,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.921795+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:31.921795+0000","last_peered":"2024-09-15T21:07:31.921795+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:31.921795+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6340,"log_dups_size":0,"ondisk_log_size":6340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49904286,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6339,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3835}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6379","reported_seq":6447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.111464+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:34.111464+0000","last_peered":"2024-09-15T21:07:34.111464+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:34.111464+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6491","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.522308+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:34.522308+0000","last_peered":"2024-09-15T21:07:34.522308+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:34.522308+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6356","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.472618+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:34.472618+0000","last_peered":"2024-09-15T21:07:34.472618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:34.472618+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6356,"log_dups_size":0,"ondisk_log_size":6356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49315840,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6543","reported_seq":6610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.077387+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:34.077387+0000","last_peered":"2024-09-15T21:07:34.077387+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:34.077387+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025164,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3863}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6561","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.030664+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:34.030664+0000","last_peered":"2024-09-15T21:07:34.030664+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:34.030664+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787596,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6409","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.404408+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:34.404408+0000","last_peered":"2024-09-15T21:07:34.404408+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:34.404408+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49498648,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6390","reported_seq":6471,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.923574+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:31.923574+0000","last_peered":"2024-09-15T21:07:31.923574+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:31.923574+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50558220,"num_objects":6172,"num_object_clones":0,"num_object_copies":24688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6388,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3837}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6343","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:34.406657+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:34.406657+0000","last_peered":"2024-09-15T21:07:34.406657+0000","last_clean":"2024-09-15T21:07:34.406657+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:34.406657+0000","last_undegraded":"2024-09-15T21:07:34.406657+0000","last_fullsized":"2024-09-15T21:07:34.406657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49291264,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6366","reported_seq":6405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.299640+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:34.299640+0000","last_peered":"2024-09-15T21:07:34.299640+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:34.299640+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49650310,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6360","reported_seq":6440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.953879+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:31.953879+0000","last_peered":"2024-09-15T21:07:31.953879+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:31.953879+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6360,"log_dups_size":0,"ondisk_log_size":6360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50305670,"num_objects":6141,"num_object_clones":0,"num_object_copies":24564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6359,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3855}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6370","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.395213+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:34.395213+0000","last_peered":"2024-09-15T21:07:34.395213+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:34.395213+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6370,"log_dups_size":0,"ondisk_log_size":6370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6455","reported_seq":6538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:31.947445+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:31.947445+0000","last_peered":"2024-09-15T21:07:31.947445+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:31.947445+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51000588,"num_objects":6226,"num_object_clones":0,"num_object_copies":24904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6453,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3945}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6440","reported_seq":6508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.138359+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:34.138359+0000","last_peered":"2024-09-15T21:07:34.138359+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:34.138359+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50042124,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6438,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":800667438,"num_objects":97743,"num_object_clones":0,"num_object_copies":390972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96669,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1469759488,"data_stored":1469759488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102582,"ondisk_log_size":102582,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755464,"kb_used_data":390568,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93616376,"statfs":{"total":96636764160,"available":95863169024,"internally_reserved":0,"allocated":399941632,"data_stored":399776643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,7,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757420,"kb_used_data":393052,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93614420,"statfs":{"total":96636764160,"available":95861166080,"internally_reserved":0,"allocated":402485248,"data_stored":402320259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":373094853},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,1,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051628,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":510404,"kb_used_data":262196,"kb_used_omap":4,"kb_used_meta":248187,"kb_avail":93861436,"statfs":{"total":96636764160,"available":96114110464,"internally_reserved":0,"allocated":268488704,"data_stored":268323715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":254144207},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747916,"kb_used_data":391420,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93623924,"statfs":{"total":96636764160,"available":95870898176,"internally_reserved":0,"allocated":400814080,"data_stored":400649091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,2,9,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399994880,"data_stored":399994880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":268128256,"data_stored":268128256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":402092032,"data_stored":402092032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":399544320,"data_stored":399544320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:37.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:37.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6343", 'reported_seq': 6388, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.478705+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:34.478705+0000', 'last_peered': '2024-09-15T21:07:34.478705+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:34.478705+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6343, 'log_dups_size': 0, 'ondisk_log_size': 6343, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49225728, 'num_objects': 6009, 'num_object_clones': 0, 'num_object_copies': 24036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6009, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6343, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3733}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.708 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:37.708 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6436", 'reported_seq': 7314, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.074458+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:34.074458+0000', 'last_peered': '2024-09-15T21:07:34.074458+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:34.074458+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6436, 'log_dups_size': 0, 'ondisk_log_size': 6436, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50099468, 'num_objects': 6116, 'num_object_clones': 0, 'num_object_copies': 24464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1975, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6434, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 407, 'num_bytes_recovered': 3317760, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6340", 'reported_seq': 6420, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:31.921795+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:31.921795+0000', 'last_peered': '2024-09-15T21:07:31.921795+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:31.921795+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6340, 'log_dups_size': 0, 'ondisk_log_size': 6340, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49904286, 'num_objects': 6093, 'num_object_clones': 0, 'num_object_copies': 24372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6093, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6339, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3835}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6379", 'reported_seq': 6447, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.111464+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:34.111464+0000', 'last_peered': '2024-09-15T21:07:34.111464+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:34.111464+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6379, 'log_dups_size': 0, 'ondisk_log_size': 6379, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49779980, 'num_objects': 6077, 'num_object_clones': 0, 'num_object_copies': 24308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6377, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3743}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6491", 'reported_seq': 6546, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.522308+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:34.522308+0000', 'last_peered': '2024-09-15T21:07:34.522308+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:34.522308+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6491, 'log_dups_size': 0, 'ondisk_log_size': 6491, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50525452, 'num_objects': 6168, 'num_object_clones': 0, 'num_object_copies': 24672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6489, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3857}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6356", 'reported_seq': 6399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.472618+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:34.472618+0000', 'last_peered': '2024-09-15T21:07:34.472618+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:34.472618+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6356, 'log_dups_size': 0, 'ondisk_log_size': 6356, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49315840, 'num_objects': 6020, 'num_object_clones': 0, 'num_object_copies': 24080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6020, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6356, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3704}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6543", 'reported_seq': 6610, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.077387+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:34.077387+0000', 'last_peered': '2024-09-15T21:07:34.077387+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:34.077387+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6543, 'log_dups_size': 0, 'ondisk_log_size': 6543, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51025164, 'num_objects': 6229, 'num_object_clones': 0, 'num_object_copies': 24916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6541, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3863}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:37.709 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6561", 'reported_seq': 6629, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.030664+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:34.030664+0000', 'last_peered': '2024-09-15T21:07:34.030664+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:34.030664+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6561, 'log_dups_size': 0, 'ondisk_log_size': 6561, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50787596, 'num_objects': 6200, 'num_object_clones': 0, 'num_object_copies': 24800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6200, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6559, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6409", 'reported_seq': 6469, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.404408+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:34.404408+0000', 'last_peered': '2024-09-15T21:07:34.404408+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:34.404408+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6409, 'log_dups_size': 0, 'ondisk_log_size': 6409, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49498648, 'num_objects': 6043, 'num_object_clones': 0, 'num_object_copies': 24172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6043, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6406, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3711}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6390", 'reported_seq': 6471, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:31.923574+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:31.923574+0000', 'last_peered': '2024-09-15T21:07:31.923574+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:31.923574+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6390, 'log_dups_size': 0, 'ondisk_log_size': 6390, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50558220, 'num_objects': 6172, 'num_object_clones': 0, 'num_object_copies': 24688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6388, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3837}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6366", 'reported_seq': 6405, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.299640+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:34.299640+0000', 'last_peered': '2024-09-15T21:07:34.299640+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:34.299640+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6366, 'log_dups_size': 0, 'ondisk_log_size': 6366, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49650310, 'num_objects': 6061, 'num_object_clones': 0, 'num_object_copies': 24244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6061, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6365, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3805}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6360", 'reported_seq': 6440, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:31.953879+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:31.953879+0000', 'last_peered': '2024-09-15T21:07:31.953879+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:31.953879+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6360, 'log_dups_size': 0, 'ondisk_log_size': 6360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50305670, 'num_objects': 6141, 'num_object_clones': 0, 'num_object_copies': 24564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6359, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:37.710 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6370", 'reported_seq': 6426, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.395213+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:34.395213+0000', 'last_peered': '2024-09-15T21:07:34.395213+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:34.395213+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6370, 'log_dups_size': 0, 'ondisk_log_size': 6370, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49657100, 'num_objects': 6062, 'num_object_clones': 0, 'num_object_copies': 24248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6368, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3783}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:37.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:37.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6455", 'reported_seq': 6538, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:31.947445+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:31.947445+0000', 'last_peered': '2024-09-15T21:07:31.947445+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:31.947445+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6455, 'log_dups_size': 0, 'ondisk_log_size': 6455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 51000588, 'num_objects': 6226, 'num_object_clones': 0, 'num_object_copies': 24904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6453, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3945}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:37.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:37.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6440", 'reported_seq': 6508, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:34.138359+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:34.138359+0000', 'last_peered': '2024-09-15T21:07:34.138359+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:34.138359+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6440, 'log_dups_size': 0, 'ondisk_log_size': 6440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50042124, 'num_objects': 6109, 'num_object_clones': 0, 'num_object_copies': 24436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6109, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6438, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:37.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:38.453 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:38.454 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34416/389696 objects degraded (8.831%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":97424,"data_bytes":798054190,"bytes_used":2841767936,"bytes_avail":383705288704,"bytes_total":386547056640,"degraded_objects":34416,"degraded_total":389696,"degraded_ratio":0.088314994251929721,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":103045,"recovering_keys_per_sec":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"read_bytes_sec":85,"write_bytes_sec":75151,"read_op_per_sec":0,"write_op_per_sec":234},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:38.475 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:38.475 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:39.071 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:39.071 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:39.095 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-15T21:07:38.004660+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":798054190,"num_objects":97424,"num_object_clones":0,"num_object_copies":389696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96988,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102901,"ondisk_log_size":102901,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2775164,"kb_used_data":1431468,"kb_used_omap":14,"kb_used_meta":1343601,"kb_avail":374712196,"statfs":{"total":386547056640,"available":383705288704,"internally_reserved":0,"allocated":1465823232,"data_stored":1465163276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1375847986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,12,18,21,9,8,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21299176,"num_objects":-2599,"num_object_clones":0,"num_object_copies":-10396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2599,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2821,"num_write_kb":881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":1236992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004332"},"pg_stats":[{"pgid":"1.f","version":"33'6343","reported_seq":6388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.478705+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:34.478705+0000","last_peered":"2024-09-15T21:07:34.478705+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:34.478705+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49225728,"num_objects":6009,"num_object_clones":0,"num_object_copies":24036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6343,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3733}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6436","reported_seq":7314,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.074458+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:34.074458+0000","last_peered":"2024-09-15T21:07:34.074458+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:34.074458+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6434,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4141}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6400","reported_seq":6480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.089112+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:37.089112+0000","last_peered":"2024-09-15T21:07:37.089112+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:37.089112+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49412766,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6399,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6379","reported_seq":6447,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.111464+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:34.111464+0000","last_peered":"2024-09-15T21:07:34.111464+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:34.111464+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6379,"log_dups_size":0,"ondisk_log_size":6379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49779980,"num_objects":6077,"num_object_clones":0,"num_object_copies":24308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3743}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6491","reported_seq":6546,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.522308+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:34.522308+0000","last_peered":"2024-09-15T21:07:34.522308+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:34.522308+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50525452,"num_objects":6168,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6489,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3857}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6356","reported_seq":6399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.472618+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:34.472618+0000","last_peered":"2024-09-15T21:07:34.472618+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:34.472618+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6356,"log_dups_size":0,"ondisk_log_size":6356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49315840,"num_objects":6020,"num_object_clones":0,"num_object_copies":24080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6356,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3704}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6543","reported_seq":6610,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.077387+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:34.077387+0000","last_peered":"2024-09-15T21:07:34.077387+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:34.077387+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6543,"log_dups_size":0,"ondisk_log_size":6543,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51025164,"num_objects":6229,"num_object_clones":0,"num_object_copies":24916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6541,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3863}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6561","reported_seq":6629,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.030664+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:34.030664+0000","last_peered":"2024-09-15T21:07:34.030664+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:34.030664+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6561,"log_dups_size":0,"ondisk_log_size":6561,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50787596,"num_objects":6200,"num_object_clones":0,"num_object_copies":24800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6559,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6409","reported_seq":6469,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.404408+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:34.404408+0000","last_peered":"2024-09-15T21:07:34.404408+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:34.404408+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6409,"log_dups_size":0,"ondisk_log_size":6409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49498648,"num_objects":6043,"num_object_clones":0,"num_object_copies":24172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6406,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3711}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6469","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.040124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:37.040124+0000","last_peered":"2024-09-15T21:07:37.040124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:37.040124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911052,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6343","reported_seq":11006,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:34.406657+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:34.406657+0000","last_peered":"2024-09-15T21:07:34.406657+0000","last_clean":"2024-09-15T21:07:34.406657+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:34.406657+0000","last_undegraded":"2024-09-15T21:07:34.406657+0000","last_fullsized":"2024-09-15T21:07:34.406657+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6343,"log_dups_size":0,"ondisk_log_size":6343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49291264,"num_objects":6017,"num_object_clones":0,"num_object_copies":24068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":454,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6366","reported_seq":6405,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.299640+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:34.299640+0000","last_peered":"2024-09-15T21:07:34.299640+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:34.299640+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6366,"log_dups_size":0,"ondisk_log_size":6366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49650310,"num_objects":6061,"num_object_clones":0,"num_object_copies":24244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6365,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3805}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6448","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.107333+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:37.107333+0000","last_peered":"2024-09-15T21:07:37.107333+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:37.107333+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49584774,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6370","reported_seq":6426,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.395213+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:34.395213+0000","last_peered":"2024-09-15T21:07:34.395213+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:34.395213+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6370,"log_dups_size":0,"ondisk_log_size":6370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49657100,"num_objects":6062,"num_object_clones":0,"num_object_copies":24248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6368,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3783}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6547","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:36.920944+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:36.920944+0000","last_peered":"2024-09-15T21:07:36.920944+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:36.920944+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6440","reported_seq":6508,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:34.138359+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:34.138359+0000","last_peered":"2024-09-15T21:07:34.138359+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:34.138359+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50042124,"num_objects":6109,"num_object_clones":0,"num_object_copies":24436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6438,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3794}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":798054190,"num_objects":97424,"num_object_clones":0,"num_object_copies":389696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97424,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":96988,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":3317760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1463721984,"data_stored":1463721984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102901,"ondisk_log_size":102901,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755464,"kb_used_data":390568,"kb_used_omap":4,"kb_used_meta":364859,"kb_avail":93616376,"statfs":{"total":96636764160,"available":95863169024,"internally_reserved":0,"allocated":399941632,"data_stored":399776643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":373616336},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,2,7,4,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739704,"kb_used_data":387816,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93632136,"statfs":{"total":96636764160,"available":95879307264,"internally_reserved":0,"allocated":397123584,"data_stored":396958595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532080,"kb_used_data":261664,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839760,"statfs":{"total":96636764160,"available":96091914240,"internally_reserved":0,"allocated":267943936,"data_stored":267778947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747916,"kb_used_data":391420,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93623924,"statfs":{"total":96636764160,"available":95870898176,"internally_reserved":0,"allocated":400814080,"data_stored":400649091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":365031118},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,3,2,9,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399994880,"data_stored":399994880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396615680,"data_stored":396615680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":399544320,"data_stored":399544320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:42.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:42.710 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:42.710 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:42.743 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-15T21:07:42.005705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":789763886,"num_objects":96412,"num_object_clones":0,"num_object_copies":385648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98000,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103913,"ondisk_log_size":103913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756272,"kb_used_data":1420908,"kb_used_omap":14,"kb_used_meta":1335281,"kb_avail":374731088,"statfs":{"total":386547056640,"available":383724634112,"internally_reserved":0,"allocated":1455009792,"data_stored":1454349836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1367328306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,17,12,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21594112,"num_objects":-2636,"num_object_clones":0,"num_object_copies":-10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"1.f","version":"33'6426","reported_seq":6470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.612898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:39.612898+0000","last_peered":"2024-09-15T21:07:39.612898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:39.612898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6528","reported_seq":7478,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.159623+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:39.159623+0000","last_peered":"2024-09-15T21:07:39.159623+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:39.159623+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345804,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6400","reported_seq":6480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.089112+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:37.089112+0000","last_peered":"2024-09-15T21:07:37.089112+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:37.089112+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49412766,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6399,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6449","reported_seq":6518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.117950+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:39.117950+0000","last_peered":"2024-09-15T21:07:39.117950+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:39.117950+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49206540,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6570","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.421156+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:39.421156+0000","last_peered":"2024-09-15T21:07:39.421156+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:39.421156+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49878284,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6446","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.443796+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:39.443796+0000","last_peered":"2024-09-15T21:07:39.443796+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:39.443796+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6635","reported_seq":6705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.034370+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:39.034370+0000","last_peered":"2024-09-15T21:07:39.034370+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:39.034370+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.171564+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:39.171564+0000","last_peered":"2024-09-15T21:07:39.171564+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:39.171564+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6485","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.484007+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:39.484007+0000","last_peered":"2024-09-15T21:07:39.484007+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:39.484007+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48876056,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6469","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.040124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:37.040124+0000","last_peered":"2024-09-15T21:07:37.040124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:37.040124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911052,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6436","reported_seq":11097,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:39.614382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:39.614382+0000","last_peered":"2024-09-15T21:07:39.614382+0000","last_clean":"2024-09-15T21:07:39.614382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:39.614382+0000","last_undegraded":"2024-09-15T21:07:39.614382+0000","last_fullsized":"2024-09-15T21:07:39.614382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6450","reported_seq":6488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.491622+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:39.491622+0000","last_peered":"2024-09-15T21:07:39.491622+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:39.491622+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962182,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6448","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.107333+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:37.107333+0000","last_peered":"2024-09-15T21:07:37.107333+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:37.107333+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49584774,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6451","reported_seq":6507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.573597+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:39.573597+0000","last_peered":"2024-09-15T21:07:39.573597+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:39.573597+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993548,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6547","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:36.920944+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:36.920944+0000","last_peered":"2024-09-15T21:07:36.920944+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:36.920944+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6528","reported_seq":6596,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.090230+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:39.090230+0000","last_peered":"2024-09-15T21:07:39.090230+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:39.090230+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49321228,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3706}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":789763886,"num_objects":96412,"num_object_clones":0,"num_object_copies":385648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98000,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1452793856,"data_stored":1452793856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103913,"ondisk_log_size":103913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737664,"kb_used_data":385260,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93634176,"statfs":{"total":96636764160,"available":95881396224,"internally_reserved":0,"allocated":394506240,"data_stored":394341251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739704,"kb_used_data":387816,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93632136,"statfs":{"total":96636764160,"available":95879307264,"internally_reserved":0,"allocated":397123584,"data_stored":396958595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532080,"kb_used_data":261664,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839760,"statfs":{"total":96636764160,"available":96091914240,"internally_reserved":0,"allocated":267943936,"data_stored":267778947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746824,"kb_used_data":386168,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93625016,"statfs":{"total":96636764160,"available":95872016384,"internally_reserved":0,"allocated":395436032,"data_stored":395271043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394588160,"data_stored":394588160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396615680,"data_stored":396615680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394022912,"data_stored":394022912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:42.744 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:43.362 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:43.363 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:43.396 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-15T21:07:42.005705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":789763886,"num_objects":96412,"num_object_clones":0,"num_object_copies":385648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98000,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103913,"ondisk_log_size":103913,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2756272,"kb_used_data":1420908,"kb_used_omap":14,"kb_used_meta":1335281,"kb_avail":374731088,"statfs":{"total":386547056640,"available":383724634112,"internally_reserved":0,"allocated":1455009792,"data_stored":1454349836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1367328306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,7,17,12,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21594112,"num_objects":-2636,"num_object_clones":0,"num_object_copies":-10544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2636,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"1.f","version":"33'6426","reported_seq":6470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.612898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:39.612898+0000","last_peered":"2024-09-15T21:07:39.612898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:39.612898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6528","reported_seq":7478,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.159623+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:39.159623+0000","last_peered":"2024-09-15T21:07:39.159623+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:39.159623+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345804,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6400","reported_seq":6480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.089112+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:37.089112+0000","last_peered":"2024-09-15T21:07:37.089112+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:37.089112+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6400,"log_dups_size":0,"ondisk_log_size":6400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49412766,"num_objects":6033,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6399,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3775}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6449","reported_seq":6518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.117950+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:39.117950+0000","last_peered":"2024-09-15T21:07:39.117950+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:39.117950+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49206540,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6570","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.421156+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:39.421156+0000","last_peered":"2024-09-15T21:07:39.421156+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:39.421156+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49878284,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6446","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.443796+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:39.443796+0000","last_peered":"2024-09-15T21:07:39.443796+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:39.443796+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6635","reported_seq":6705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.034370+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:39.034370+0000","last_peered":"2024-09-15T21:07:39.034370+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:39.034370+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.171564+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:39.171564+0000","last_peered":"2024-09-15T21:07:39.171564+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:39.171564+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6485","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.484007+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:39.484007+0000","last_peered":"2024-09-15T21:07:39.484007+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:39.484007+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48876056,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6469","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.040124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:37.040124+0000","last_peered":"2024-09-15T21:07:37.040124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:37.040124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49911052,"num_objects":6093,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6467,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3758}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6436","reported_seq":11097,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:39.614382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:39.614382+0000","last_peered":"2024-09-15T21:07:39.614382+0000","last_clean":"2024-09-15T21:07:39.614382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:39.614382+0000","last_undegraded":"2024-09-15T21:07:39.614382+0000","last_fullsized":"2024-09-15T21:07:39.614382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6450","reported_seq":6488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.491622+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:39.491622+0000","last_peered":"2024-09-15T21:07:39.491622+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:39.491622+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962182,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6448","reported_seq":6527,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:37.107333+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:37.107333+0000","last_peered":"2024-09-15T21:07:37.107333+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:37.107333+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49584774,"num_objects":6053,"num_object_clones":0,"num_object_copies":24212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3767}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6451","reported_seq":6507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.573597+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:39.573597+0000","last_peered":"2024-09-15T21:07:39.573597+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:39.573597+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993548,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6547","reported_seq":6630,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:36.920944+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:36.920944+0000","last_peered":"2024-09-15T21:07:36.920944+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:36.920944+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50246924,"num_objects":6134,"num_object_clones":0,"num_object_copies":24536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6528","reported_seq":6596,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.090230+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:39.090230+0000","last_peered":"2024-09-15T21:07:39.090230+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:39.090230+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49321228,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3706}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":789763886,"num_objects":96412,"num_object_clones":0,"num_object_copies":385648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96412,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98000,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1452793856,"data_stored":1452793856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103913,"ondisk_log_size":103913,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737664,"kb_used_data":385260,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93634176,"statfs":{"total":96636764160,"available":95881396224,"internally_reserved":0,"allocated":394506240,"data_stored":394341251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739704,"kb_used_data":387816,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93632136,"statfs":{"total":96636764160,"available":95879307264,"internally_reserved":0,"allocated":397123584,"data_stored":396958595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051629,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":532080,"kb_used_data":261664,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93839760,"statfs":{"total":96636764160,"available":96091914240,"internally_reserved":0,"allocated":267943936,"data_stored":267778947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746824,"kb_used_data":386168,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93625016,"statfs":{"total":96636764160,"available":95872016384,"internally_reserved":0,"allocated":395436032,"data_stored":395271043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394588160,"data_stored":394588160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267567104,"data_stored":267567104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":396615680,"data_stored":396615680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394022912,"data_stored":394022912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6426", 'reported_seq': 6470, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.612898+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:39.612898+0000', 'last_peered': '2024-09-15T21:07:39.612898+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:39.612898+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6426, 'log_dups_size': 0, 'ondisk_log_size': 6426, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48545792, 'num_objects': 5926, 'num_object_clones': 0, 'num_object_copies': 23704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6426, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3650}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:43.397 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6528", 'reported_seq': 7478, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.159623+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:39.159623+0000', 'last_peered': '2024-09-15T21:07:39.159623+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:39.159623+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6528, 'log_dups_size': 0, 'ondisk_log_size': 6528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49345804, 'num_objects': 6024, 'num_object_clones': 0, 'num_object_copies': 24096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1939, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6024, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6526, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 443, 'num_bytes_recovered': 3612672, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4085}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6400", 'reported_seq': 6480, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:37.089112+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:37.089112+0000', 'last_peered': '2024-09-15T21:07:37.089112+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:37.089112+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6400, 'log_dups_size': 0, 'ondisk_log_size': 6400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49412766, 'num_objects': 6033, 'num_object_clones': 0, 'num_object_copies': 24132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6033, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6399, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3775}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6449", 'reported_seq': 6518, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.117950+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:39.117950+0000', 'last_peered': '2024-09-15T21:07:39.117950+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:39.117950+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6449, 'log_dups_size': 0, 'ondisk_log_size': 6449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49206540, 'num_objects': 6007, 'num_object_clones': 0, 'num_object_copies': 24028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6570", 'reported_seq': 6626, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.421156+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:39.421156+0000', 'last_peered': '2024-09-15T21:07:39.421156+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:39.421156+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6570, 'log_dups_size': 0, 'ondisk_log_size': 6570, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49878284, 'num_objects': 6089, 'num_object_clones': 0, 'num_object_copies': 24356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6089, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6568, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3778}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6446", 'reported_seq': 6490, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.443796+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:39.443796+0000', 'last_peered': '2024-09-15T21:07:39.443796+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:39.443796+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6446, 'log_dups_size': 0, 'ondisk_log_size': 6446, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48578560, 'num_objects': 5930, 'num_object_clones': 0, 'num_object_copies': 23720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6446, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3614}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:43.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6635", 'reported_seq': 6705, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.034370+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:39.034370+0000', 'last_peered': '2024-09-15T21:07:39.034370+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:39.034370+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6635, 'log_dups_size': 0, 'ondisk_log_size': 6635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50271500, 'num_objects': 6137, 'num_object_clones': 0, 'num_object_copies': 24548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6633, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3771}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6645", 'reported_seq': 6714, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.171564+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:39.171564+0000', 'last_peered': '2024-09-15T21:07:39.171564+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:39.171564+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6645, 'log_dups_size': 0, 'ondisk_log_size': 6645, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50099468, 'num_objects': 6116, 'num_object_clones': 0, 'num_object_copies': 24464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6643, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3642}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6485", 'reported_seq': 6545, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.484007+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:39.484007+0000', 'last_peered': '2024-09-15T21:07:39.484007+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:39.484007+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6485, 'log_dups_size': 0, 'ondisk_log_size': 6485, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48876056, 'num_objects': 5967, 'num_object_clones': 0, 'num_object_copies': 23868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6482, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3635}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6469", 'reported_seq': 6550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:37.040124+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:37.040124+0000', 'last_peered': '2024-09-15T21:07:37.040124+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:37.040124+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6469, 'log_dups_size': 0, 'ondisk_log_size': 6469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49911052, 'num_objects': 6093, 'num_object_clones': 0, 'num_object_copies': 24372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6467, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3758}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:43.399 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6450", 'reported_seq': 6488, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.491622+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:39.491622+0000', 'last_peered': '2024-09-15T21:07:39.491622+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:39.491622+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6450, 'log_dups_size': 0, 'ondisk_log_size': 6450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48962182, 'num_objects': 5977, 'num_object_clones': 0, 'num_object_copies': 23908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5977, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6449, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3721}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6448", 'reported_seq': 6527, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:37.107333+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:37.107333+0000', 'last_peered': '2024-09-15T21:07:37.107333+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:37.107333+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6448, 'log_dups_size': 0, 'ondisk_log_size': 6448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49584774, 'num_objects': 6053, 'num_object_clones': 0, 'num_object_copies': 24212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6053, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6447, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3767}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6451", 'reported_seq': 6507, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.573597+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:39.573597+0000', 'last_peered': '2024-09-15T21:07:39.573597+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:39.573597+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6451, 'log_dups_size': 0, 'ondisk_log_size': 6451, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48993548, 'num_objects': 5981, 'num_object_clones': 0, 'num_object_copies': 23924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6449, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6547", 'reported_seq': 6630, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:36.920944+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:36.920944+0000', 'last_peered': '2024-09-15T21:07:36.920944+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:36.920944+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6547, 'log_dups_size': 0, 'ondisk_log_size': 6547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 50246924, 'num_objects': 6134, 'num_object_clones': 0, 'num_object_copies': 24536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6134, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6545, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3853}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:43.400 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6528", 'reported_seq': 6596, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:39.090230+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:39.090230+0000', 'last_peered': '2024-09-15T21:07:39.090230+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:39.090230+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6528, 'log_dups_size': 0, 'ondisk_log_size': 6528, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49321228, 'num_objects': 6021, 'num_object_clones': 0, 'num_object_copies': 24084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6526, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3706}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:43.401 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:44.218 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:44.218 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34380/385648 objects degraded (8.915%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":329,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":96412,"data_bytes":789763886,"bytes_used":2822422528,"bytes_avail":383724634112,"bytes_total":386547056640,"degraded_objects":34380,"degraded_total":385648,"degraded_ratio":0.089148653694560848,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":52548,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:44.259 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:44.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:44.896 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:44.897 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:44.924 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-15T21:07:44.006392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":787076910,"num_objects":96084,"num_object_clones":0,"num_object_copies":384336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98328,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104241,"ondisk_log_size":104241,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758896,"kb_used_data":1415184,"kb_used_omap":14,"kb_used_meta":1343601,"kb_avail":374728464,"statfs":{"total":386547056640,"available":383721947136,"internally_reserved":0,"allocated":1449148416,"data_stored":1448488460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1375847986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,14,9,11,16,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24281088,"num_objects":-2964,"num_object_clones":0,"num_object_copies":-11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003695"},"pg_stats":[{"pgid":"1.f","version":"33'6426","reported_seq":6470,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.612898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:39.612898+0000","last_peered":"2024-09-15T21:07:39.612898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:39.612898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48545792,"num_objects":5926,"num_object_clones":0,"num_object_copies":23704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6426,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6528","reported_seq":7478,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.159623+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:39.159623+0000","last_peered":"2024-09-15T21:07:39.159623+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:39.159623+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49345804,"num_objects":6024,"num_object_clones":0,"num_object_copies":24096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1939,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6494","reported_seq":6574,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:42.055906+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:42.055906+0000","last_peered":"2024-09-15T21:07:42.055906+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:42.055906+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6494,"log_dups_size":0,"ondisk_log_size":6494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48642718,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6493,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6449","reported_seq":6518,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.117950+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:39.117950+0000","last_peered":"2024-09-15T21:07:39.117950+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:39.117950+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49206540,"num_objects":6007,"num_object_clones":0,"num_object_copies":24028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6570","reported_seq":6626,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.421156+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:39.421156+0000","last_peered":"2024-09-15T21:07:39.421156+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:39.421156+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6570,"log_dups_size":0,"ondisk_log_size":6570,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49878284,"num_objects":6089,"num_object_clones":0,"num_object_copies":24356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6568,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3778}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6446","reported_seq":6490,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.443796+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:39.443796+0000","last_peered":"2024-09-15T21:07:39.443796+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:39.443796+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6446,"log_dups_size":0,"ondisk_log_size":6446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48578560,"num_objects":5930,"num_object_clones":0,"num_object_copies":23720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6446,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6635","reported_seq":6705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.034370+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:39.034370+0000","last_peered":"2024-09-15T21:07:39.034370+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:39.034370+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6635,"log_dups_size":0,"ondisk_log_size":6635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50271500,"num_objects":6137,"num_object_clones":0,"num_object_copies":24548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6633,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3771}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6645","reported_seq":6714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.171564+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:39.171564+0000","last_peered":"2024-09-15T21:07:39.171564+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:39.171564+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6645,"log_dups_size":0,"ondisk_log_size":6645,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50099468,"num_objects":6116,"num_object_clones":0,"num_object_copies":24464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6643,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3642}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6485","reported_seq":6545,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.484007+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:39.484007+0000","last_peered":"2024-09-15T21:07:39.484007+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:39.484007+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6485,"log_dups_size":0,"ondisk_log_size":6485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48876056,"num_objects":5967,"num_object_clones":0,"num_object_copies":23868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6482,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6546","reported_seq":6627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:42.146834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:42.146834+0000","last_peered":"2024-09-15T21:07:42.146834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:42.146834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49280268,"num_objects":6016,"num_object_clones":0,"num_object_copies":24064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6544,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3681}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6436","reported_seq":11097,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:39.614382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:39.614382+0000","last_peered":"2024-09-15T21:07:39.614382+0000","last_clean":"2024-09-15T21:07:39.614382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:39.614382+0000","last_undegraded":"2024-09-15T21:07:39.614382+0000","last_fullsized":"2024-09-15T21:07:39.614382+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48529408,"num_objects":5924,"num_object_clones":0,"num_object_copies":23696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6450","reported_seq":6488,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.491622+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:39.491622+0000","last_peered":"2024-09-15T21:07:39.491622+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:39.491622+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6450,"log_dups_size":0,"ondisk_log_size":6450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48962182,"num_objects":5977,"num_object_clones":0,"num_object_copies":23908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3721}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6520","reported_seq":6600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:42.126388+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:42.126388+0000","last_peered":"2024-09-15T21:07:42.126388+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:42.126388+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6520,"log_dups_size":0,"ondisk_log_size":6520,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48994950,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3695}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6451","reported_seq":6507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.573597+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:39.573597+0000","last_peered":"2024-09-15T21:07:39.573597+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:39.573597+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48993548,"num_objects":5981,"num_object_clones":0,"num_object_copies":23924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6449,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6632","reported_seq":6715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:42.012132+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:42.012132+0000","last_peered":"2024-09-15T21:07:42.012132+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:42.012132+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49550604,"num_objects":6049,"num_object_clones":0,"num_object_copies":24196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6630,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3768}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6528","reported_seq":6596,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:39.090230+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:39.090230+0000","last_peered":"2024-09-15T21:07:39.090230+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:39.090230+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49321228,"num_objects":6021,"num_object_clones":0,"num_object_copies":24084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6526,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3706}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":787076910,"num_objects":96084,"num_object_clones":0,"num_object_copies":384336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98328,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":465,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1446809600,"data_stored":1446809600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104241,"ondisk_log_size":104241,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737664,"kb_used_data":385260,"kb_used_omap":4,"kb_used_meta":352379,"kb_avail":93634176,"statfs":{"total":96636764160,"available":95881396224,"internally_reserved":0,"allocated":394506240,"data_stored":394341251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":360836816},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738712,"kb_used_data":382636,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93633128,"statfs":{"total":96636764160,"available":95880323072,"internally_reserved":0,"allocated":391819264,"data_stored":391654275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051630,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":535696,"kb_used_data":261120,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93836144,"statfs":{"total":96636764160,"available":96088211456,"internally_reserved":0,"allocated":267386880,"data_stored":267221891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,6,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746824,"kb_used_data":386168,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93625016,"statfs":{"total":96636764160,"available":95872016384,"internally_reserved":0,"allocated":395436032,"data_stored":395271043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":369290958},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,7,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":394588160,"data_stored":394588160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":267038720,"data_stored":267038720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":391159808,"data_stored":391159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":394022912,"data_stored":394022912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:47.926 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:48.505 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:48.506 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:48.531 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-15T21:07:48.008288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":776197934,"num_objects":94756,"num_object_clones":0,"num_object_copies":379024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99656,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105569,"ondisk_log_size":105569,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742752,"kb_used_data":1399028,"kb_used_omap":14,"kb_used_meta":1343601,"kb_avail":374744608,"statfs":{"total":386547056640,"available":383738478592,"internally_reserved":0,"allocated":1432604672,"data_stored":1431944716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1375847986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,11,16,7,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24469504,"num_objects":-2987,"num_object_clones":0,"num_object_copies":-11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004170"},"pg_stats":[{"pgid":"1.f","version":"33'6504","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.547461+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:44.547461+0000","last_peered":"2024-09-15T21:07:44.547461+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:44.547461+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6600","reported_seq":7635,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.198947+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:44.198947+0000","last_peered":"2024-09-15T21:07:44.198947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:44.198947+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48755980,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":485,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6585","reported_seq":6665,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.174932+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:47.174932+0000","last_peered":"2024-09-15T21:07:47.174932+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:47.174932+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47897246,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6584,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6547","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.121244+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:44.121244+0000","last_peered":"2024-09-15T21:07:44.121244+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:44.121244+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48403724,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6653","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.631052+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:44.631052+0000","last_peered":"2024-09-15T21:07:44.631052+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:44.631052+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49198348,"num_objects":6006,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6651,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6517","reported_seq":6562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.703200+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:44.703200+0000","last_peered":"2024-09-15T21:07:44.703200+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:44.703200+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6517,"log_dups_size":0,"ondisk_log_size":6517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6517,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6722","reported_seq":6792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.147910+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:44.147910+0000","last_peered":"2024-09-15T21:07:44.147910+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:44.147910+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49558796,"num_objects":6050,"num_object_clones":0,"num_object_copies":24200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6724","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.136751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:44.136751+0000","last_peered":"2024-09-15T21:07:44.136751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:44.136751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49452300,"num_objects":6037,"num_object_clones":0,"num_object_copies":24148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6567","reported_seq":6627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.641625+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:44.641625+0000","last_peered":"2024-09-15T21:07:44.641625+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:44.641625+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48204312,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6639","reported_seq":6720,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.232844+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:47.232844+0000","last_peered":"2024-09-15T21:07:47.232844+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:47.232844+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48518412,"num_objects":5923,"num_object_clones":0,"num_object_copies":23692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6519","reported_seq":11182,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:44.664842+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:44.664842+0000","last_peered":"2024-09-15T21:07:44.664842+0000","last_clean":"2024-09-15T21:07:44.664842+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:44.664842+0000","last_undegraded":"2024-09-15T21:07:44.664842+0000","last_fullsized":"2024-09-15T21:07:44.664842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6519,"log_dups_size":0,"ondisk_log_size":6519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47849472,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6514","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.626325+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:44.626325+0000","last_peered":"2024-09-15T21:07:44.626325+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:44.626325+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48437894,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6513,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6604","reported_seq":6684,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.116679+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:47.116679+0000","last_peered":"2024-09-15T21:07:47.116679+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:47.116679+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306822,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6538","reported_seq":6595,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.660000+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:44.660000+0000","last_peered":"2024-09-15T21:07:44.660000+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:44.660000+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6538,"log_dups_size":0,"ondisk_log_size":6538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280844,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6726","reported_seq":6807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.161009+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:47.161009+0000","last_peered":"2024-09-15T21:07:47.161009+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:47.161009+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6610","reported_seq":6679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.230098+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:44.230098+0000","last_peered":"2024-09-15T21:07:44.230098+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:44.230098+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48649484,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":776197934,"num_objects":94756,"num_object_clones":0,"num_object_copies":379024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99656,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1430081536,"data_stored":1430081536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105569,"ondisk_log_size":105569,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736668,"kb_used_data":380080,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93635172,"statfs":{"total":96636764160,"available":95882416128,"internally_reserved":0,"allocated":389201920,"data_stored":389036931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737768,"kb_used_data":377556,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93634072,"statfs":{"total":96636764160,"available":95881289728,"internally_reserved":0,"allocated":386617344,"data_stored":386452355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522724,"kb_used_data":260628,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93849116,"statfs":{"total":96636764160,"available":96101494784,"internally_reserved":0,"allocated":266883072,"data_stored":266718083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745592,"kb_used_data":380764,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93626248,"statfs":{"total":96636764160,"available":95873277952,"internally_reserved":0,"allocated":389902336,"data_stored":389737347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389091328,"data_stored":389091328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266551296,"data_stored":266551296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385785856,"data_stored":385785856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":388653056,"data_stored":388653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:48.532 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:49.170 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:49.171 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:49.197 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-15T21:07:48.008288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":776197934,"num_objects":94756,"num_object_clones":0,"num_object_copies":379024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99656,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105569,"ondisk_log_size":105569,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742752,"kb_used_data":1399028,"kb_used_omap":14,"kb_used_meta":1343601,"kb_avail":374744608,"statfs":{"total":386547056640,"available":383738478592,"internally_reserved":0,"allocated":1432604672,"data_stored":1431944716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1375847986},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,11,16,7,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24469504,"num_objects":-2987,"num_object_clones":0,"num_object_copies":-11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-78,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004170"},"pg_stats":[{"pgid":"1.f","version":"33'6504","reported_seq":6550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.547461+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:44.547461+0000","last_peered":"2024-09-15T21:07:44.547461+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:44.547461+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6504,"log_dups_size":0,"ondisk_log_size":6504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6504,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3572}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6600","reported_seq":7635,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.198947+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:44.198947+0000","last_peered":"2024-09-15T21:07:44.198947+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:44.198947+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6600,"log_dups_size":0,"ondisk_log_size":6600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48755980,"num_objects":5952,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1897,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6598,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":485,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6585","reported_seq":6665,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.174932+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:47.174932+0000","last_peered":"2024-09-15T21:07:47.174932+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:47.174932+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47897246,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6584,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6547","reported_seq":6616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.121244+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:44.121244+0000","last_peered":"2024-09-15T21:07:44.121244+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:44.121244+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48403724,"num_objects":5909,"num_object_clones":0,"num_object_copies":23636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6545,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3575}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6653","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.631052+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:44.631052+0000","last_peered":"2024-09-15T21:07:44.631052+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:44.631052+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49198348,"num_objects":6006,"num_object_clones":0,"num_object_copies":24024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6651,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3695}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6517","reported_seq":6562,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.703200+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:44.703200+0000","last_peered":"2024-09-15T21:07:44.703200+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:44.703200+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6517,"log_dups_size":0,"ondisk_log_size":6517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47996928,"num_objects":5859,"num_object_clones":0,"num_object_copies":23436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6517,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3543}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6722","reported_seq":6792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.147910+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:44.147910+0000","last_peered":"2024-09-15T21:07:44.147910+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:44.147910+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6722,"log_dups_size":0,"ondisk_log_size":6722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49558796,"num_objects":6050,"num_object_clones":0,"num_object_copies":24200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6720,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6724","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.136751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:44.136751+0000","last_peered":"2024-09-15T21:07:44.136751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:44.136751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49452300,"num_objects":6037,"num_object_clones":0,"num_object_copies":24148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6722,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6567","reported_seq":6627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.641625+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:44.641625+0000","last_peered":"2024-09-15T21:07:44.641625+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:44.641625+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6567,"log_dups_size":0,"ondisk_log_size":6567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48204312,"num_objects":5885,"num_object_clones":0,"num_object_copies":23540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6564,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6639","reported_seq":6720,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.232844+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:47.232844+0000","last_peered":"2024-09-15T21:07:47.232844+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:47.232844+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48518412,"num_objects":5923,"num_object_clones":0,"num_object_copies":23692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6519","reported_seq":11182,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:44.664842+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:44.664842+0000","last_peered":"2024-09-15T21:07:44.664842+0000","last_clean":"2024-09-15T21:07:44.664842+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:44.664842+0000","last_undegraded":"2024-09-15T21:07:44.664842+0000","last_fullsized":"2024-09-15T21:07:44.664842+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6519,"log_dups_size":0,"ondisk_log_size":6519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47849472,"num_objects":5841,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6514","reported_seq":6554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.626325+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:44.626325+0000","last_peered":"2024-09-15T21:07:44.626325+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:44.626325+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6514,"log_dups_size":0,"ondisk_log_size":6514,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48437894,"num_objects":5913,"num_object_clones":0,"num_object_copies":23652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6513,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3657}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6604","reported_seq":6684,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.116679+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:47.116679+0000","last_peered":"2024-09-15T21:07:47.116679+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:47.116679+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306822,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6538","reported_seq":6595,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.660000+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:44.660000+0000","last_peered":"2024-09-15T21:07:44.660000+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:44.660000+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6538,"log_dups_size":0,"ondisk_log_size":6538,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48280844,"num_objects":5894,"num_object_clones":0,"num_object_copies":23576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3615}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6726","reported_seq":6807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.161009+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:47.161009+0000","last_peered":"2024-09-15T21:07:47.161009+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:47.161009+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6610","reported_seq":6679,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:44.230098+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:44.230098+0000","last_peered":"2024-09-15T21:07:44.230098+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:44.230098+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6610,"log_dups_size":0,"ondisk_log_size":6610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48649484,"num_objects":5939,"num_object_clones":0,"num_object_copies":23756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6608,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3624}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":776197934,"num_objects":94756,"num_object_clones":0,"num_object_copies":379024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94756,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99656,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":3956736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1430081536,"data_stored":1430081536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105569,"ondisk_log_size":105569,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736668,"kb_used_data":380080,"kb_used_omap":4,"kb_used_meta":356539,"kb_avail":93635172,"statfs":{"total":96636764160,"available":95882416128,"internally_reserved":0,"allocated":389201920,"data_stored":389036931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":365096656},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,3,0,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737768,"kb_used_data":377556,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93634072,"statfs":{"total":96636764160,"available":95881289728,"internally_reserved":0,"allocated":386617344,"data_stored":386452355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522724,"kb_used_data":260628,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93849116,"statfs":{"total":96636764160,"available":96101494784,"internally_reserved":0,"allocated":266883072,"data_stored":266718083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745592,"kb_used_data":380764,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93626248,"statfs":{"total":96636764160,"available":95873277952,"internally_reserved":0,"allocated":389902336,"data_stored":389737347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":373550798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389091328,"data_stored":389091328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266551296,"data_stored":266551296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385785856,"data_stored":385785856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":388653056,"data_stored":388653056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:49.198 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:49.198 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6504", 'reported_seq': 6550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.547461+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:44.547461+0000', 'last_peered': '2024-09-15T21:07:44.547461+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:44.547461+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6504, 'log_dups_size': 0, 'ondisk_log_size': 6504, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47906816, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6504, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3572}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6600", 'reported_seq': 7635, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.198947+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:44.198947+0000', 'last_peered': '2024-09-15T21:07:44.198947+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:44.198947+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6600, 'log_dups_size': 0, 'ondisk_log_size': 6600, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48755980, 'num_objects': 5952, 'num_object_clones': 0, 'num_object_copies': 23808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1897, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6598, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 485, 'num_bytes_recovered': 3956736, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6585", 'reported_seq': 6665, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:47.174932+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:47.174932+0000', 'last_peered': '2024-09-15T21:07:47.174932+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:47.174932+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6585, 'log_dups_size': 0, 'ondisk_log_size': 6585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47897246, 'num_objects': 5848, 'num_object_clones': 0, 'num_object_copies': 23392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5848, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6584, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6547", 'reported_seq': 6616, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.121244+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:44.121244+0000', 'last_peered': '2024-09-15T21:07:44.121244+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:44.121244+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6547, 'log_dups_size': 0, 'ondisk_log_size': 6547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48403724, 'num_objects': 5909, 'num_object_clones': 0, 'num_object_copies': 23636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5909, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6545, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6653", 'reported_seq': 6709, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.631052+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:44.631052+0000', 'last_peered': '2024-09-15T21:07:44.631052+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:44.631052+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6653, 'log_dups_size': 0, 'ondisk_log_size': 6653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49198348, 'num_objects': 6006, 'num_object_clones': 0, 'num_object_copies': 24024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6651, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3695}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:49.199 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6517", 'reported_seq': 6562, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.703200+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:44.703200+0000', 'last_peered': '2024-09-15T21:07:44.703200+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:44.703200+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6517, 'log_dups_size': 0, 'ondisk_log_size': 6517, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47996928, 'num_objects': 5859, 'num_object_clones': 0, 'num_object_copies': 23436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5859, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6517, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3543}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6722", 'reported_seq': 6792, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.147910+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:44.147910+0000', 'last_peered': '2024-09-15T21:07:44.147910+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:44.147910+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6722, 'log_dups_size': 0, 'ondisk_log_size': 6722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49558796, 'num_objects': 6050, 'num_object_clones': 0, 'num_object_copies': 24200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6720, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3684}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6724", 'reported_seq': 6793, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.136751+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:44.136751+0000', 'last_peered': '2024-09-15T21:07:44.136751+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:44.136751+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6724, 'log_dups_size': 0, 'ondisk_log_size': 6724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 49452300, 'num_objects': 6037, 'num_object_clones': 0, 'num_object_copies': 24148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6037, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6722, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6567", 'reported_seq': 6627, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.641625+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:44.641625+0000', 'last_peered': '2024-09-15T21:07:44.641625+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:44.641625+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6567, 'log_dups_size': 0, 'ondisk_log_size': 6567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48204312, 'num_objects': 5885, 'num_object_clones': 0, 'num_object_copies': 23540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6564, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6639", 'reported_seq': 6720, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:47.232844+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:47.232844+0000', 'last_peered': '2024-09-15T21:07:47.232844+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:47.232844+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6639, 'log_dups_size': 0, 'ondisk_log_size': 6639, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48518412, 'num_objects': 5923, 'num_object_clones': 0, 'num_object_copies': 23692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6637, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3588}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6514", 'reported_seq': 6554, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.626325+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:44.626325+0000', 'last_peered': '2024-09-15T21:07:44.626325+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:44.626325+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6514, 'log_dups_size': 0, 'ondisk_log_size': 6514, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48437894, 'num_objects': 5913, 'num_object_clones': 0, 'num_object_copies': 23652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5913, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6513, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.200 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6604", 'reported_seq': 6684, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:47.116679+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:47.116679+0000', 'last_peered': '2024-09-15T21:07:47.116679+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:47.116679+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6604, 'log_dups_size': 0, 'ondisk_log_size': 6604, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48306822, 'num_objects': 5897, 'num_object_clones': 0, 'num_object_copies': 23588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6603, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3611}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6538", 'reported_seq': 6595, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.660000+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:44.660000+0000', 'last_peered': '2024-09-15T21:07:44.660000+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:44.660000+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6538, 'log_dups_size': 0, 'ondisk_log_size': 6538, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48280844, 'num_objects': 5894, 'num_object_clones': 0, 'num_object_copies': 23576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5894, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6536, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3615}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6726", 'reported_seq': 6807, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:47.161009+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:47.161009+0000', 'last_peered': '2024-09-15T21:07:47.161009+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:47.161009+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6726, 'log_dups_size': 0, 'ondisk_log_size': 6726, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48780556, 'num_objects': 5955, 'num_object_clones': 0, 'num_object_copies': 23820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6724, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3674}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:49.201 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6610", 'reported_seq': 6679, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:44.230098+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:44.230098+0000', 'last_peered': '2024-09-15T21:07:44.230098+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:44.230098+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6610, 'log_dups_size': 0, 'ondisk_log_size': 6610, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48649484, 'num_objects': 5939, 'num_object_clones': 0, 'num_object_copies': 23756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5939, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6608, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:49.201 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:49.993 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:49.993 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34338/379024 objects degraded (9.060%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":335,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":94756,"data_bytes":776197934,"bytes_used":2808578048,"bytes_avail":383738478592,"bytes_total":386547056640,"degraded_objects":34338,"degraded_total":379024,"degraded_ratio":0.0905958461733294,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":53229,"recovering_keys_per_sec":0,"num_objects_recovered":78,"num_bytes_recovered":638976,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":248},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:50.026 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:50.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:50.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:50.607 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:50.628 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-09-15T21:07:50.011109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":768120622,"num_objects":93770,"num_object_clones":0,"num_object_copies":375080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100642,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106555,"ondisk_log_size":106555,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742032,"kb_used_data":1388600,"kb_used_omap":14,"kb_used_meta":1353329,"kb_avail":374745328,"statfs":{"total":386547056640,"available":383739215872,"internally_reserved":0,"allocated":1421926400,"data_stored":1421266444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1385809458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,17,12,7,6,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29933568,"num_objects":-3654,"num_object_clones":0,"num_object_copies":-14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":1015808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006449"},"pg_stats":[{"pgid":"1.f","version":"33'6569","reported_seq":6615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.700244+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:49.700244+0000","last_peered":"2024-09-15T21:07:49.700244+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:49.700244+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47374336,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6694","reported_seq":7821,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.394108+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:49.394108+0000","last_peered":"2024-09-15T21:07:49.394108+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:49.394108+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985932,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6585","reported_seq":6665,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.174932+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:47.174932+0000","last_peered":"2024-09-15T21:07:47.174932+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:47.174932+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6585,"log_dups_size":0,"ondisk_log_size":6585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47897246,"num_objects":5848,"num_object_clones":0,"num_object_copies":23392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6584,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3590}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.188141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:49.188141+0000","last_peered":"2024-09-15T21:07:49.188141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:49.188141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6737","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.602906+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:49.602906+0000","last_peered":"2024-09-15T21:07:49.602906+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:49.602906+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6602","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.746548+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:49.746548+0000","last_peered":"2024-09-15T21:07:49.746548+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:49.746548+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47300608,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6817","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.201361+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:49.201361+0000","last_peered":"2024-09-15T21:07:49.201361+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:49.201361+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":6880,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.317171+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:49.317171+0000","last_peered":"2024-09-15T21:07:49.317171+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:49.317171+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48739596,"num_objects":5950,"num_object_clones":0,"num_object_copies":23800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6653","reported_seq":6713,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.743916+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:49.743916+0000","last_peered":"2024-09-15T21:07:49.743916+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:49.743916+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47499800,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6639","reported_seq":6720,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.232844+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:47.232844+0000","last_peered":"2024-09-15T21:07:47.232844+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:47.232844+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6639,"log_dups_size":0,"ondisk_log_size":6639,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48518412,"num_objects":5923,"num_object_clones":0,"num_object_copies":23692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6637,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6594","reported_seq":11257,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:49.647098+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:49.647098+0000","last_peered":"2024-09-15T21:07:49.647098+0000","last_clean":"2024-09-15T21:07:49.647098+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:49.647098+0000","last_undegraded":"2024-09-15T21:07:49.647098+0000","last_fullsized":"2024-09-15T21:07:49.647098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47235072,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":705,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6581","reported_seq":6621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.492277+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:49.492277+0000","last_peered":"2024-09-15T21:07:49.492277+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:49.492277+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6581,"log_dups_size":0,"ondisk_log_size":6581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889030,"num_objects":5846,"num_object_clones":0,"num_object_copies":23384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6580,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6604","reported_seq":6684,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.116679+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:47.116679+0000","last_peered":"2024-09-15T21:07:47.116679+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:47.116679+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48306822,"num_objects":5897,"num_object_clones":0,"num_object_copies":23588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6603,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3611}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6617","reported_seq":6674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.639721+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:49.639721+0000","last_peered":"2024-09-15T21:07:49.639721+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:49.639721+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47633676,"num_objects":5815,"num_object_clones":0,"num_object_copies":23260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6726","reported_seq":6807,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:47.161009+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:47.161009+0000","last_peered":"2024-09-15T21:07:47.161009+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:47.161009+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6726,"log_dups_size":0,"ondisk_log_size":6726,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3674}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6702","reported_seq":6771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.321818+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:49.321818+0000","last_peered":"2024-09-15T21:07:49.321818+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:49.321818+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895820,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6700,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":768120622,"num_objects":93770,"num_object_clones":0,"num_object_copies":375080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93770,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100642,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1419202560,"data_stored":1419202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106555,"ondisk_log_size":106555,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":374836,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":383832064,"data_stored":383667075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":387837648},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737768,"kb_used_data":377556,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93634072,"statfs":{"total":96636764160,"available":95881289728,"internally_reserved":0,"allocated":386617344,"data_stored":386452355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,6,1,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051631,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":522724,"kb_used_data":260628,"kb_used_omap":4,"kb_used_meta":262075,"kb_avail":93849116,"statfs":{"total":96636764160,"available":96101494784,"internally_reserved":0,"allocated":266883072,"data_stored":266718083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":268365519},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727928,"kb_used_data":375580,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93643912,"statfs":{"total":96636764160,"available":95891365888,"internally_reserved":0,"allocated":384593920,"data_stored":384428931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383614976,"data_stored":383614976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266551296,"data_stored":266551296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":385785856,"data_stored":385785856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383250432,"data_stored":383250432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:53.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:54.215 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:54.215 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:54.243 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T21:07:54.014127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":765474606,"num_objects":93447,"num_object_clones":0,"num_object_copies":373788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100965,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106878,"ondisk_log_size":106878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762792,"kb_used_data":1382996,"kb_used_omap":14,"kb_used_meta":1379697,"kb_avail":374724568,"statfs":{"total":386547056640,"available":383717957632,"internally_reserved":0,"allocated":1416187904,"data_stored":1415527948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1412810290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,12,5,11,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24289280,"num_objects":-2965,"num_object_clones":0,"num_object_copies":-11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008422"},"pg_stats":[{"pgid":"1.f","version":"33'6569","reported_seq":6615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.700244+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:49.700244+0000","last_peered":"2024-09-15T21:07:49.700244+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:49.700244+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47374336,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6694","reported_seq":7821,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.394108+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:49.394108+0000","last_peered":"2024-09-15T21:07:49.394108+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:49.394108+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985932,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6673","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.264026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:52.264026+0000","last_peered":"2024-09-15T21:07:52.264026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:52.264026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47176350,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6672,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.188141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:49.188141+0000","last_peered":"2024-09-15T21:07:49.188141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:49.188141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6737","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.602906+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:49.602906+0000","last_peered":"2024-09-15T21:07:49.602906+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:49.602906+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6602","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.746548+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:49.746548+0000","last_peered":"2024-09-15T21:07:49.746548+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:49.746548+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47300608,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6817","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.201361+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:49.201361+0000","last_peered":"2024-09-15T21:07:49.201361+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:49.201361+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":6880,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.317171+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:49.317171+0000","last_peered":"2024-09-15T21:07:49.317171+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:49.317171+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48739596,"num_objects":5950,"num_object_clones":0,"num_object_copies":23800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6653","reported_seq":6713,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.743916+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:49.743916+0000","last_peered":"2024-09-15T21:07:49.743916+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:49.743916+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47499800,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6717","reported_seq":6797,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.239189+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:52.239189+0000","last_peered":"2024-09-15T21:07:52.239189+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:52.239189+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47879436,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6594","reported_seq":11257,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:49.647098+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:49.647098+0000","last_peered":"2024-09-15T21:07:49.647098+0000","last_clean":"2024-09-15T21:07:49.647098+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:49.647098+0000","last_undegraded":"2024-09-15T21:07:49.647098+0000","last_fullsized":"2024-09-15T21:07:49.647098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47235072,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":705,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6581","reported_seq":6621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.492277+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:49.492277+0000","last_peered":"2024-09-15T21:07:49.492277+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:49.492277+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6581,"log_dups_size":0,"ondisk_log_size":6581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889030,"num_objects":5846,"num_object_clones":0,"num_object_copies":23384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6580,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6689","reported_seq":6769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.107205+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:52.107205+0000","last_peered":"2024-09-15T21:07:52.107205+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:52.107205+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47610502,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6617","reported_seq":6674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.639721+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:49.639721+0000","last_peered":"2024-09-15T21:07:49.639721+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:49.639721+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47633676,"num_objects":5815,"num_object_clones":0,"num_object_copies":23260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6798","reported_seq":6881,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.224375+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:52.224375+0000","last_peered":"2024-09-15T21:07:52.224375+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:52.224375+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190732,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3602}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6702","reported_seq":6771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.321818+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:49.321818+0000","last_peered":"2024-09-15T21:07:49.321818+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:49.321818+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895820,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6700,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":765474606,"num_objects":93447,"num_object_clones":0,"num_object_copies":373788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100965,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413390336,"data_stored":1413390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106878,"ondisk_log_size":106878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":374836,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":383832064,"data_stored":383667075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":387837648},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754836,"kb_used_data":372420,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93617004,"statfs":{"total":96636764160,"available":95863812096,"internally_reserved":0,"allocated":381358080,"data_stored":381193091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526416,"kb_used_data":260160,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93845424,"statfs":{"total":96636764160,"available":96097714176,"internally_reserved":0,"allocated":266403840,"data_stored":266238851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727928,"kb_used_data":375580,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93643912,"statfs":{"total":96636764160,"available":95891365888,"internally_reserved":0,"allocated":384593920,"data_stored":384428931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383614976,"data_stored":383614976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266043392,"data_stored":266043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380481536,"data_stored":380481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383250432,"data_stored":383250432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:54.244 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:54.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:54.873 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:54.893 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2024-09-15T21:07:54.014127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":765474606,"num_objects":93447,"num_object_clones":0,"num_object_copies":373788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100965,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106878,"ondisk_log_size":106878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762792,"kb_used_data":1382996,"kb_used_omap":14,"kb_used_meta":1379697,"kb_avail":374724568,"statfs":{"total":386547056640,"available":383717957632,"internally_reserved":0,"allocated":1416187904,"data_stored":1415527948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1412810290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,12,5,11,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24289280,"num_objects":-2965,"num_object_clones":0,"num_object_copies":-11860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2965,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008422"},"pg_stats":[{"pgid":"1.f","version":"33'6569","reported_seq":6615,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.700244+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:49.700244+0000","last_peered":"2024-09-15T21:07:49.700244+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:49.700244+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47374336,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6694","reported_seq":7821,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.394108+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:49.394108+0000","last_peered":"2024-09-15T21:07:49.394108+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:49.394108+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47985932,"num_objects":5858,"num_object_clones":0,"num_object_copies":23432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6692,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":531,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":4007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6673","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.264026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:52.264026+0000","last_peered":"2024-09-15T21:07:52.264026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:52.264026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47176350,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6672,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6624","reported_seq":6693,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.188141+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:49.188141+0000","last_peered":"2024-09-15T21:07:49.188141+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:49.188141+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6624,"log_dups_size":0,"ondisk_log_size":6624,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47772940,"num_objects":5832,"num_object_clones":0,"num_object_copies":23328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6622,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3498}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6737","reported_seq":6793,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.602906+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:49.602906+0000","last_peered":"2024-09-15T21:07:49.602906+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:49.602906+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6737,"log_dups_size":0,"ondisk_log_size":6737,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48510220,"num_objects":5922,"num_object_clones":0,"num_object_copies":23688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3611}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6602","reported_seq":6647,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.746548+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:49.746548+0000","last_peered":"2024-09-15T21:07:49.746548+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:49.746548+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6602,"log_dups_size":0,"ondisk_log_size":6602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47300608,"num_objects":5774,"num_object_clones":0,"num_object_copies":23096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6817","reported_seq":6887,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.201361+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:49.201361+0000","last_peered":"2024-09-15T21:07:49.201361+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:49.201361+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6817,"log_dups_size":0,"ondisk_log_size":6817,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48780556,"num_objects":5955,"num_object_clones":0,"num_object_copies":23820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6815,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3589}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6811","reported_seq":6880,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.317171+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:49.317171+0000","last_peered":"2024-09-15T21:07:49.317171+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:49.317171+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6811,"log_dups_size":0,"ondisk_log_size":6811,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48739596,"num_objects":5950,"num_object_clones":0,"num_object_copies":23800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3476}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6653","reported_seq":6713,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.743916+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:49.743916+0000","last_peered":"2024-09-15T21:07:49.743916+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:49.743916+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47499800,"num_objects":5799,"num_object_clones":0,"num_object_copies":23196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6650,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6717","reported_seq":6797,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.239189+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:52.239189+0000","last_peered":"2024-09-15T21:07:52.239189+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:52.239189+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47879436,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6594","reported_seq":11257,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:49.647098+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:49.647098+0000","last_peered":"2024-09-15T21:07:49.647098+0000","last_clean":"2024-09-15T21:07:49.647098+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:49.647098+0000","last_undegraded":"2024-09-15T21:07:49.647098+0000","last_fullsized":"2024-09-15T21:07:49.647098+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6594,"log_dups_size":0,"ondisk_log_size":6594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47235072,"num_objects":5766,"num_object_clones":0,"num_object_copies":23064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":705,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6581","reported_seq":6621,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.492277+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:49.492277+0000","last_peered":"2024-09-15T21:07:49.492277+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:49.492277+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6581,"log_dups_size":0,"ondisk_log_size":6581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47889030,"num_objects":5846,"num_object_clones":0,"num_object_copies":23384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6580,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3590}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6689","reported_seq":6769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.107205+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:52.107205+0000","last_peered":"2024-09-15T21:07:52.107205+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:52.107205+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47610502,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6617","reported_seq":6674,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.639721+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:49.639721+0000","last_peered":"2024-09-15T21:07:49.639721+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:49.639721+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6617,"log_dups_size":0,"ondisk_log_size":6617,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47633676,"num_objects":5815,"num_object_clones":0,"num_object_copies":23260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6615,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3536}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6798","reported_seq":6881,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.224375+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:52.224375+0000","last_peered":"2024-09-15T21:07:52.224375+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:52.224375+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190732,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3602}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6702","reported_seq":6771,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:49.321818+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:49.321818+0000","last_peered":"2024-09-15T21:07:49.321818+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:49.321818+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6702,"log_dups_size":0,"ondisk_log_size":6702,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47895820,"num_objects":5847,"num_object_clones":0,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6700,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":765474606,"num_objects":93447,"num_object_clones":0,"num_object_copies":373788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34292,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93447,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100965,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":4333568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1413390336,"data_stored":1413390336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106878,"ondisk_log_size":106878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753612,"kb_used_data":374836,"kb_used_omap":4,"kb_used_meta":378747,"kb_avail":93618228,"statfs":{"total":96636764160,"available":95865065472,"internally_reserved":0,"allocated":383832064,"data_stored":383667075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":387837648},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754836,"kb_used_data":372420,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93617004,"statfs":{"total":96636764160,"available":95863812096,"internally_reserved":0,"allocated":381358080,"data_stored":381193091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526416,"kb_used_data":260160,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93845424,"statfs":{"total":96636764160,"available":96097714176,"internally_reserved":0,"allocated":266403840,"data_stored":266238851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727928,"kb_used_data":375580,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93643912,"statfs":{"total":96636764160,"available":95891365888,"internally_reserved":0,"allocated":384593920,"data_stored":384428931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":360771278},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,7,2,2,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383614976,"data_stored":383614976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266043392,"data_stored":266043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380481536,"data_stored":380481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":383250432,"data_stored":383250432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6569", 'reported_seq': 6615, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.700244+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:49.700244+0000', 'last_peered': '2024-09-15T21:07:49.700244+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:49.700244+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6569, 'log_dups_size': 0, 'ondisk_log_size': 6569, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47374336, 'num_objects': 5783, 'num_object_clones': 0, 'num_object_copies': 23132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6569, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6694", 'reported_seq': 7821, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.394108+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:49.394108+0000', 'last_peered': '2024-09-15T21:07:49.394108+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:49.394108+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6694, 'log_dups_size': 0, 'ondisk_log_size': 6694, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47985932, 'num_objects': 5858, 'num_object_clones': 0, 'num_object_copies': 23432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1851, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5858, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6692, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 531, 'num_bytes_recovered': 4333568, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 4007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6673", 'reported_seq': 6753, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:52.264026+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:52.264026+0000', 'last_peered': '2024-09-15T21:07:52.264026+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:52.264026+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6673, 'log_dups_size': 0, 'ondisk_log_size': 6673, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47176350, 'num_objects': 5760, 'num_object_clones': 0, 'num_object_copies': 23040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5760, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6672, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3502}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:07:54.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6624", 'reported_seq': 6693, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.188141+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:49.188141+0000', 'last_peered': '2024-09-15T21:07:49.188141+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:49.188141+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6624, 'log_dups_size': 0, 'ondisk_log_size': 6624, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47772940, 'num_objects': 5832, 'num_object_clones': 0, 'num_object_copies': 23328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6622, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3498}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6737", 'reported_seq': 6793, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.602906+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:49.602906+0000', 'last_peered': '2024-09-15T21:07:49.602906+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:49.602906+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6737, 'log_dups_size': 0, 'ondisk_log_size': 6737, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48510220, 'num_objects': 5922, 'num_object_clones': 0, 'num_object_copies': 23688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5922, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6735, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3611}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6602", 'reported_seq': 6647, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.746548+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:49.746548+0000', 'last_peered': '2024-09-15T21:07:49.746548+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:49.746548+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6602, 'log_dups_size': 0, 'ondisk_log_size': 6602, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47300608, 'num_objects': 5774, 'num_object_clones': 0, 'num_object_copies': 23096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5774, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6602, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3458}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6817", 'reported_seq': 6887, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.201361+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:49.201361+0000', 'last_peered': '2024-09-15T21:07:49.201361+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:49.201361+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6817, 'log_dups_size': 0, 'ondisk_log_size': 6817, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48780556, 'num_objects': 5955, 'num_object_clones': 0, 'num_object_copies': 23820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5955, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6815, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3589}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6811", 'reported_seq': 6880, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.317171+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:49.317171+0000', 'last_peered': '2024-09-15T21:07:49.317171+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:49.317171+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6811, 'log_dups_size': 0, 'ondisk_log_size': 6811, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48739596, 'num_objects': 5950, 'num_object_clones': 0, 'num_object_copies': 23800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5950, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6809, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3476}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:54.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6653", 'reported_seq': 6713, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.743916+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:49.743916+0000', 'last_peered': '2024-09-15T21:07:49.743916+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:49.743916+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6653, 'log_dups_size': 0, 'ondisk_log_size': 6653, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47499800, 'num_objects': 5799, 'num_object_clones': 0, 'num_object_copies': 23196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5799, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6650, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3467}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6717", 'reported_seq': 6797, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:52.239189+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:52.239189+0000', 'last_peered': '2024-09-15T21:07:52.239189+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:52.239189+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6717, 'log_dups_size': 0, 'ondisk_log_size': 6717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47879436, 'num_objects': 5845, 'num_object_clones': 0, 'num_object_copies': 23380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5845, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6715, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3510}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6581", 'reported_seq': 6621, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.492277+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:49.492277+0000', 'last_peered': '2024-09-15T21:07:49.492277+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:49.492277+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6581, 'log_dups_size': 0, 'ondisk_log_size': 6581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47889030, 'num_objects': 5846, 'num_object_clones': 0, 'num_object_copies': 23384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5846, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6580, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6689", 'reported_seq': 6769, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:52.107205+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:52.107205+0000', 'last_peered': '2024-09-15T21:07:52.107205+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:52.107205+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6689, 'log_dups_size': 0, 'ondisk_log_size': 6689, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47610502, 'num_objects': 5812, 'num_object_clones': 0, 'num_object_copies': 23248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6688, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6617", 'reported_seq': 6674, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.639721+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:49.639721+0000', 'last_peered': '2024-09-15T21:07:49.639721+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:49.639721+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6617, 'log_dups_size': 0, 'ondisk_log_size': 6617, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47633676, 'num_objects': 5815, 'num_object_clones': 0, 'num_object_copies': 23260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6615, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3536}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:07:54.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6798", 'reported_seq': 6881, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:52.224375+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:52.224375+0000', 'last_peered': '2024-09-15T21:07:52.224375+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:52.224375+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6798, 'log_dups_size': 0, 'ondisk_log_size': 6798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 48190732, 'num_objects': 5883, 'num_object_clones': 0, 'num_object_copies': 23532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6796, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3602}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:07:54.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:07:54.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6702", 'reported_seq': 6771, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:49.321818+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:49.321818+0000', 'last_peered': '2024-09-15T21:07:49.321818+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:49.321818+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6702, 'log_dups_size': 0, 'ondisk_log_size': 6702, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47895820, 'num_objects': 5847, 'num_object_clones': 0, 'num_object_copies': 23388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5847, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6700, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:07:54.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:07:55.637 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:55.637 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34292/373788 objects degraded (9.174%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":341,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":93447,"data_bytes":765474606,"bytes_used":2829099008,"bytes_avail":383717957632,"bytes_total":386547056640,"degraded_objects":34292,"degraded_total":373788,"degraded_ratio":0.091741842969811763,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":60032,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":246},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:07:55.669 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:07:55.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:56.304 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:56.305 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:56.330 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-15T21:07:56.014743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":757274414,"num_objects":92446,"num_object_clones":0,"num_object_copies":369784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101966,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107879,"ondisk_log_size":107879,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2778824,"kb_used_data":1372652,"kb_used_omap":14,"kb_used_meta":1406065,"kb_avail":374708536,"statfs":{"total":386547056640,"available":383701540864,"internally_reserved":0,"allocated":1405595648,"data_stored":1404935692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1439811122},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,12,4,12,9,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29802496,"num_objects":-3638,"num_object_clones":0,"num_object_copies":-14552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3638,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":130,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008351"},"pg_stats":[{"pgid":"1.f","version":"33'6662","reported_seq":6708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.776185+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:54.776185+0000","last_peered":"2024-09-15T21:07:54.776185+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:54.776185+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46612480,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6662,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6770","reported_seq":7980,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.461100+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:54.461100+0000","last_peered":"2024-09-15T21:07:54.461100+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:54.461100+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47363340,"num_objects":5782,"num_object_clones":0,"num_object_copies":23128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6673","reported_seq":6753,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.264026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:52.264026+0000","last_peered":"2024-09-15T21:07:52.264026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:52.264026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6673,"log_dups_size":0,"ondisk_log_size":6673,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47176350,"num_objects":5760,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5760,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6672,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3502}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6725","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.367728+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:54.367728+0000","last_peered":"2024-09-15T21:07:54.367728+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:54.367728+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46945548,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6822","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.591851+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:54.591851+0000","last_peered":"2024-09-15T21:07:54.591851+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:54.591851+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47813900,"num_objects":5837,"num_object_clones":0,"num_object_copies":23348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6661","reported_seq":6706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.613981+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:54.613981+0000","last_peered":"2024-09-15T21:07:54.613981+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:54.613981+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46817280,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6902","reported_seq":6972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.295355+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:54.295355+0000","last_peered":"2024-09-15T21:07:54.295355+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:54.295355+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48084236,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6900,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6885","reported_seq":6954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.259062+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:54.259062+0000","last_peered":"2024-09-15T21:07:54.259062+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:54.259062+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48133388,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6731","reported_seq":6790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.790134+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:54.790134+0000","last_peered":"2024-09-15T21:07:54.790134+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:54.790134+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6731,"log_dups_size":0,"ondisk_log_size":6731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46860824,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6717","reported_seq":6797,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.239189+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:52.239189+0000","last_peered":"2024-09-15T21:07:52.239189+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:52.239189+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6717,"log_dups_size":0,"ondisk_log_size":6717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47879436,"num_objects":5845,"num_object_clones":0,"num_object_copies":23380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6715,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6688","reported_seq":11351,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:54.605355+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:54.605355+0000","last_peered":"2024-09-15T21:07:54.605355+0000","last_clean":"2024-09-15T21:07:54.605355+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:54.605355+0000","last_undegraded":"2024-09-15T21:07:54.605355+0000","last_fullsized":"2024-09-15T21:07:54.605355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":799,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6669","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.780553+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:54.780553+0000","last_peered":"2024-09-15T21:07:54.780553+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:54.780553+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47168134,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6689","reported_seq":6769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.107205+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:52.107205+0000","last_peered":"2024-09-15T21:07:52.107205+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:52.107205+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47610502,"num_objects":5812,"num_object_clones":0,"num_object_copies":23248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3526}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6696","reported_seq":6751,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.739309+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:54.739309+0000","last_peered":"2024-09-15T21:07:54.739309+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:54.739309+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46986508,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3457}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6798","reported_seq":6881,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:52.224375+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:52.224375+0000","last_peered":"2024-09-15T21:07:52.224375+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:52.224375+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6798,"log_dups_size":0,"ondisk_log_size":6798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48190732,"num_objects":5883,"num_object_clones":0,"num_object_copies":23532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6796,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3602}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6791","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.343389+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:54.343389+0000","last_peered":"2024-09-15T21:07:54.343389+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:54.343389+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6791,"log_dups_size":0,"ondisk_log_size":6791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47166732,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":757274414,"num_objects":92446,"num_object_clones":0,"num_object_copies":369784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":101966,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1402703872,"data_stored":1402703872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107879,"ondisk_log_size":107879,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752584,"kb_used_data":369632,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93619256,"statfs":{"total":96636764160,"available":95866118144,"internally_reserved":0,"allocated":378503168,"data_stored":378338179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":392097488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754836,"kb_used_data":372420,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":93617004,"statfs":{"total":96636764160,"available":95863812096,"internally_reserved":0,"allocated":381358080,"data_stored":381193091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,6,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051632,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526416,"kb_used_data":260160,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93845424,"statfs":{"total":96636764160,"available":96097714176,"internally_reserved":0,"allocated":266403840,"data_stored":266238851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744988,"kb_used_data":370440,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93626852,"statfs":{"total":96636764160,"available":95873896448,"internally_reserved":0,"allocated":379330560,"data_stored":379165571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":383512270},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,1,4,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378277888,"data_stored":378277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":266043392,"data_stored":266043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":380481536,"data_stored":380481536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377901056,"data_stored":377901056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:59.334 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:07:59.930 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:07:59.930 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:07:59.950 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-15T21:07:58.015412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":754702126,"num_objects":92132,"num_object_clones":0,"num_object_copies":368528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102280,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108193,"ondisk_log_size":108193,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2768912,"kb_used_data":1366888,"kb_used_omap":14,"kb_used_meta":1401905,"kb_avail":374718448,"statfs":{"total":386547056640,"available":383711690752,"internally_reserved":0,"allocated":1399693312,"data_stored":1399033356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1435551282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,13,9,9,10,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24461312,"num_objects":-2986,"num_object_clones":0,"num_object_copies":-11944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2986,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008070"},"pg_stats":[{"pgid":"1.f","version":"33'6662","reported_seq":6708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.776185+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:54.776185+0000","last_peered":"2024-09-15T21:07:54.776185+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:54.776185+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6662,"log_dups_size":0,"ondisk_log_size":6662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46612480,"num_objects":5690,"num_object_clones":0,"num_object_copies":22760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6662,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3414}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6770","reported_seq":7980,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.461100+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:54.461100+0000","last_peered":"2024-09-15T21:07:54.461100+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:54.461100+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6770,"log_dups_size":0,"ondisk_log_size":6770,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47363340,"num_objects":5782,"num_object_clones":0,"num_object_copies":23128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1809,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6768,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.415091+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:57.415091+0000","last_peered":"2024-09-15T21:07:57.415091+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:57.415091+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46561950,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6725","reported_seq":6794,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.367728+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:54.367728+0000","last_peered":"2024-09-15T21:07:54.367728+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:54.367728+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6725,"log_dups_size":0,"ondisk_log_size":6725,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46945548,"num_objects":5731,"num_object_clones":0,"num_object_copies":22924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6723,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3397}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6822","reported_seq":6878,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.591851+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:54.591851+0000","last_peered":"2024-09-15T21:07:54.591851+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:54.591851+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47813900,"num_objects":5837,"num_object_clones":0,"num_object_copies":23348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6820,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6661","reported_seq":6706,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.613981+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:54.613981+0000","last_peered":"2024-09-15T21:07:54.613981+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:54.613981+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6661,"log_dups_size":0,"ondisk_log_size":6661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46817280,"num_objects":5715,"num_object_clones":0,"num_object_copies":22860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6661,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3399}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6902","reported_seq":6972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.295355+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:54.295355+0000","last_peered":"2024-09-15T21:07:54.295355+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:54.295355+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48084236,"num_objects":5870,"num_object_clones":0,"num_object_copies":23480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6900,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3504}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6885","reported_seq":6954,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.259062+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:54.259062+0000","last_peered":"2024-09-15T21:07:54.259062+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:54.259062+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48133388,"num_objects":5876,"num_object_clones":0,"num_object_copies":23504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6883,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3402}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6731","reported_seq":6790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.790134+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:54.790134+0000","last_peered":"2024-09-15T21:07:54.790134+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:54.790134+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6731,"log_dups_size":0,"ondisk_log_size":6731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46860824,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6728,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3389}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6801","reported_seq":6882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.423194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:57.423194+0000","last_peered":"2024-09-15T21:07:57.423194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:57.423194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3426}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6688","reported_seq":11351,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:54.605355+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:54.605355+0000","last_peered":"2024-09-15T21:07:54.605355+0000","last_clean":"2024-09-15T21:07:54.605355+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:54.605355+0000","last_undegraded":"2024-09-15T21:07:54.605355+0000","last_fullsized":"2024-09-15T21:07:54.605355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":799,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6669","reported_seq":6709,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.780553+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:54.780553+0000","last_peered":"2024-09-15T21:07:54.780553+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:54.780553+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47168134,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6668,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3502}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6780","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.422945+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:57.422945+0000","last_peered":"2024-09-15T21:07:57.422945+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:57.422945+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46865030,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3435}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6696","reported_seq":6751,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.739309+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:54.739309+0000","last_peered":"2024-09-15T21:07:54.739309+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:54.739309+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6696,"log_dups_size":0,"ondisk_log_size":6696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46986508,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6694,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3457}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6862","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.377612+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:57.377612+0000","last_peered":"2024-09-15T21:07:57.377612+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:57.377612+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666444,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6860,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6791","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:54.343389+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:54.343389+0000","last_peered":"2024-09-15T21:07:54.343389+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:54.343389+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6791,"log_dups_size":0,"ondisk_log_size":6791,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47166732,"num_objects":5758,"num_object_clones":0,"num_object_copies":23032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6789,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3443}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":754702126,"num_objects":92132,"num_object_clones":0,"num_object_copies":368528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92132,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":102280,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":595,"num_bytes_recovered":4677632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1396854784,"data_stored":1396854784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108193,"ondisk_log_size":108193,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752584,"kb_used_data":369632,"kb_used_omap":4,"kb_used_meta":382907,"kb_avail":93619256,"statfs":{"total":96636764160,"available":95866118144,"internally_reserved":0,"allocated":378503168,"data_stored":378338179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":392097488},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1,0,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741264,"kb_used_data":367156,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93630576,"statfs":{"total":96636764160,"available":95877709824,"internally_reserved":0,"allocated":375967744,"data_stored":375802755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530076,"kb_used_data":259660,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93841764,"statfs":{"total":96636764160,"available":96093966336,"internally_reserved":0,"allocated":265891840,"data_stored":265726851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":744988,"kb_used_data":370440,"kb_used_omap":4,"kb_used_meta":374523,"kb_avail":93626852,"statfs":{"total":96636764160,"available":95873896448,"internally_reserved":0,"allocated":379330560,"data_stored":379165571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":383512270},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,1,4,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":378277888,"data_stored":378277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265560064,"data_stored":265560064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375115776,"data_stored":375115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":377901056,"data_stored":377901056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:07:59.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:00.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:00.597 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:00.625 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-15T21:08:00.016316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":746706734,"num_objects":91156,"num_object_clones":0,"num_object_copies":364624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103256,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109169,"ondisk_log_size":109169,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789748,"kb_used_data":1355804,"kb_used_omap":14,"kb_used_meta":1433841,"kb_avail":374697612,"statfs":{"total":386547056640,"available":383690354688,"internally_reserved":0,"allocated":1388343296,"data_stored":1387683340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1468253746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,19,8,22,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29491200,"num_objects":-3600,"num_object_clones":0,"num_object_copies":-14400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3600,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008028"},"pg_stats":[{"pgid":"1.f","version":"33'6736","reported_seq":6782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.619361+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:59.619361+0000","last_peered":"2024-09-15T21:07:59.619361+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:59.619361+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6736,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6845","reported_seq":8145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.513912+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:59.513912+0000","last_peered":"2024-09-15T21:07:59.513912+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:59.513912+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748940,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.415091+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:57.415091+0000","last_peered":"2024-09-15T21:07:57.415091+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:57.415091+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46561950,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6810","reported_seq":6879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.527250+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:59.527250+0000","last_peered":"2024-09-15T21:07:59.527250+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:59.527250+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6810,"log_dups_size":0,"ondisk_log_size":6810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46249228,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6903","reported_seq":6959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.847582+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:59.847582+0000","last_peered":"2024-09-15T21:07:59.847582+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:59.847582+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150348,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6749","reported_seq":6792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.817742+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:59.817742+0000","last_peered":"2024-09-15T21:07:59.817742+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:59.817742+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6749,"log_dups_size":0,"ondisk_log_size":6749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6986","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.399455+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:59.399455+0000","last_peered":"2024-09-15T21:07:59.399455+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:59.399455+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6978","reported_seq":7047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.397305+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:59.397305+0000","last_peered":"2024-09-15T21:07:59.397305+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:59.397305+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6800","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.782919+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:59.782919+0000","last_peered":"2024-09-15T21:07:59.782919+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:59.782919+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46295576,"num_objects":5652,"num_object_clones":0,"num_object_copies":22608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6801","reported_seq":6882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.423194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:57.423194+0000","last_peered":"2024-09-15T21:07:57.423194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:57.423194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3426}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6777","reported_seq":11440,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:59.740127+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:59.740127+0000","last_peered":"2024-09-15T21:07:59.740127+0000","last_clean":"2024-09-15T21:07:59.740127+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:59.740127+0000","last_undegraded":"2024-09-15T21:07:59.740127+0000","last_fullsized":"2024-09-15T21:07:59.740127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45735936,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6745","reported_seq":6785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.748566+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:59.748566+0000","last_peered":"2024-09-15T21:07:59.748566+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:59.748566+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46545542,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6780","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.422945+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:57.422945+0000","last_peered":"2024-09-15T21:07:57.422945+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:57.422945+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46865030,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3435}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6772","reported_seq":6829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.854603+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:59.854603+0000","last_peered":"2024-09-15T21:07:59.854603+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:59.854603+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6772,"log_dups_size":0,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6862","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.377612+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:57.377612+0000","last_peered":"2024-09-15T21:07:57.377612+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:57.377612+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666444,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6860,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6877","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.517592+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:59.517592+0000","last_peered":"2024-09-15T21:07:59.517592+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:59.517592+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":746706734,"num_objects":91156,"num_object_clones":0,"num_object_copies":364624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103256,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1386110976,"data_stored":1386110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109169,"ondisk_log_size":109169,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752984,"kb_used_data":364484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93618856,"statfs":{"total":96636764160,"available":95865708544,"internally_reserved":0,"allocated":373231616,"data_stored":373066627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741264,"kb_used_data":367156,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93630576,"statfs":{"total":96636764160,"available":95877709824,"internally_reserved":0,"allocated":375967744,"data_stored":375802755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530076,"kb_used_data":259660,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93841764,"statfs":{"total":96636764160,"available":96093966336,"internally_reserved":0,"allocated":265891840,"data_stored":265726851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765424,"kb_used_data":364504,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93606416,"statfs":{"total":96636764160,"available":95852969984,"internally_reserved":0,"allocated":373252096,"data_stored":373087107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372887552,"data_stored":372887552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265560064,"data_stored":265560064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375115776,"data_stored":375115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372547584,"data_stored":372547584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6736", 'reported_seq': 6782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.619361+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:07:59.619361+0000', 'last_peered': '2024-09-15T21:07:59.619361+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:07:59.619361+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6736, 'log_dups_size': 0, 'ondisk_log_size': 6736, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46006272, 'num_objects': 5616, 'num_object_clones': 0, 'num_object_copies': 22464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6736, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6845", 'reported_seq': 8145, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.513912+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:07:59.513912+0000', 'last_peered': '2024-09-15T21:07:59.513912+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:07:59.513912+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6845, 'log_dups_size': 0, 'ondisk_log_size': 6845, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46748940, 'num_objects': 5707, 'num_object_clones': 0, 'num_object_copies': 22828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1764, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6843, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 618, 'num_bytes_recovered': 5046272, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3943}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:00.626 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6748", 'reported_seq': 6828, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:57.415091+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:07:57.415091+0000', 'last_peered': '2024-09-15T21:07:57.415091+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:07:57.415091+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6748, 'log_dups_size': 0, 'ondisk_log_size': 6748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46561950, 'num_objects': 5685, 'num_object_clones': 0, 'num_object_copies': 22740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5685, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6747, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6810", 'reported_seq': 6879, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.527250+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:07:59.527250+0000', 'last_peered': '2024-09-15T21:07:59.527250+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:07:59.527250+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6810, 'log_dups_size': 0, 'ondisk_log_size': 6810, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46249228, 'num_objects': 5646, 'num_object_clones': 0, 'num_object_copies': 22584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5646, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6808, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3312}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6903", 'reported_seq': 6959, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.847582+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:07:59.847582+0000', 'last_peered': '2024-09-15T21:07:59.847582+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:07:59.847582+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6903, 'log_dups_size': 0, 'ondisk_log_size': 6903, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47150348, 'num_objects': 5756, 'num_object_clones': 0, 'num_object_copies': 23024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6901, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3445}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6749", 'reported_seq': 6792, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.817742+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:07:59.817742+0000', 'last_peered': '2024-09-15T21:07:59.817742+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:07:59.817742+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6749, 'log_dups_size': 0, 'ondisk_log_size': 6749, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46096384, 'num_objects': 5627, 'num_object_clones': 0, 'num_object_copies': 22508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5627, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6749, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3311}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'6986", 'reported_seq': 7056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.399455+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:07:59.399455+0000', 'last_peered': '2024-09-15T21:07:59.399455+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:07:59.399455+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6986, 'log_dups_size': 0, 'ondisk_log_size': 6986, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47396108, 'num_objects': 5786, 'num_object_clones': 0, 'num_object_copies': 23144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6984, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3420}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:00.627 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'6978", 'reported_seq': 7047, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.397305+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:07:59.397305+0000', 'last_peered': '2024-09-15T21:07:59.397305+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:07:59.397305+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6978, 'log_dups_size': 0, 'ondisk_log_size': 6978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47371532, 'num_objects': 5783, 'num_object_clones': 0, 'num_object_copies': 23132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6976, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6800", 'reported_seq': 6860, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.782919+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:07:59.782919+0000', 'last_peered': '2024-09-15T21:07:59.782919+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:07:59.782919+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6800, 'log_dups_size': 0, 'ondisk_log_size': 6800, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46295576, 'num_objects': 5652, 'num_object_clones': 0, 'num_object_copies': 22608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6797, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3320}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6801", 'reported_seq': 6882, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:57.423194+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:07:57.423194+0000', 'last_peered': '2024-09-15T21:07:57.423194+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:07:57.423194+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6801, 'log_dups_size': 0, 'ondisk_log_size': 6801, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47191308, 'num_objects': 5761, 'num_object_clones': 0, 'num_object_copies': 23044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6799, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3426}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6745", 'reported_seq': 6785, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.748566+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:07:59.748566+0000', 'last_peered': '2024-09-15T21:07:59.748566+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:07:59.748566+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6745, 'log_dups_size': 0, 'ondisk_log_size': 6745, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46545542, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 22728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6744, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6780", 'reported_seq': 6858, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:57.422945+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:07:57.422945+0000', 'last_peered': '2024-09-15T21:07:57.422945+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:07:57.422945+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6780, 'log_dups_size': 0, 'ondisk_log_size': 6780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46865030, 'num_objects': 5721, 'num_object_clones': 0, 'num_object_copies': 22884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5721, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6779, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3435}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:00.628 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6772", 'reported_seq': 6829, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.854603+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:07:59.854603+0000', 'last_peered': '2024-09-15T21:07:59.854603+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:07:59.854603+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6772, 'log_dups_size': 0, 'ondisk_log_size': 6772, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46363916, 'num_objects': 5660, 'num_object_clones': 0, 'num_object_copies': 22640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6770, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:00.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:00.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6862", 'reported_seq': 6945, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:57.377612+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:07:57.377612+0000', 'last_peered': '2024-09-15T21:07:57.377612+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:07:57.377612+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6862, 'log_dups_size': 0, 'ondisk_log_size': 6862, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 47666444, 'num_objects': 5819, 'num_object_clones': 0, 'num_object_copies': 23276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5819, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6860, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3538}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:00.629 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:00.629 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6877", 'reported_seq': 6944, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:07:59.517592+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:07:59.517592+0000', 'last_peered': '2024-09-15T21:07:59.517592+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:07:59.517592+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6877, 'log_dups_size': 0, 'ondisk_log_size': 6877, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46462220, 'num_objects': 5672, 'num_object_clones': 0, 'num_object_copies': 22688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6875, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3357}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:00.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:01.377 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:01.378 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34205/364624 objects degraded (9.381%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":347,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":91156,"data_bytes":746706734,"bytes_used":2856701952,"bytes_avail":383690354688,"bytes_total":386547056640,"degraded_objects":34205,"degraded_total":364624,"degraded_ratio":0.093808964851463422,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":90733,"recovering_keys_per_sec":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":299},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:01.402 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:01.402 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:02.059 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:02.059 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:02.086 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-15T21:08:02.017035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":746706734,"num_objects":91156,"num_object_clones":0,"num_object_copies":364624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103256,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109169,"ondisk_log_size":109169,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2789748,"kb_used_data":1355804,"kb_used_omap":14,"kb_used_meta":1433841,"kb_avail":374697612,"statfs":{"total":386547056640,"available":383690354688,"internally_reserved":0,"allocated":1388343296,"data_stored":1387683340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1468253746},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,19,8,22,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21413888,"num_objects":-2614,"num_object_clones":0,"num_object_copies":-10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2614,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":712704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005925"},"pg_stats":[{"pgid":"1.f","version":"33'6736","reported_seq":6782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.619361+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:59.619361+0000","last_peered":"2024-09-15T21:07:59.619361+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:59.619361+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6736,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6845","reported_seq":8145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.513912+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:59.513912+0000","last_peered":"2024-09-15T21:07:59.513912+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:59.513912+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748940,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6748","reported_seq":6828,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.415091+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:07:57.415091+0000","last_peered":"2024-09-15T21:07:57.415091+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:07:57.415091+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46561950,"num_objects":5685,"num_object_clones":0,"num_object_copies":22740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6810","reported_seq":6879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.527250+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:59.527250+0000","last_peered":"2024-09-15T21:07:59.527250+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:59.527250+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6810,"log_dups_size":0,"ondisk_log_size":6810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46249228,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6903","reported_seq":6959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.847582+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:59.847582+0000","last_peered":"2024-09-15T21:07:59.847582+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:59.847582+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150348,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6749","reported_seq":6792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.817742+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:59.817742+0000","last_peered":"2024-09-15T21:07:59.817742+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:59.817742+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6749,"log_dups_size":0,"ondisk_log_size":6749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6986","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.399455+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:59.399455+0000","last_peered":"2024-09-15T21:07:59.399455+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:59.399455+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6978","reported_seq":7047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.397305+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:59.397305+0000","last_peered":"2024-09-15T21:07:59.397305+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:59.397305+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6800","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.782919+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:59.782919+0000","last_peered":"2024-09-15T21:07:59.782919+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:59.782919+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46295576,"num_objects":5652,"num_object_clones":0,"num_object_copies":22608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6801","reported_seq":6882,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.423194+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:07:57.423194+0000","last_peered":"2024-09-15T21:07:57.423194+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:07:57.423194+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6801,"log_dups_size":0,"ondisk_log_size":6801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47191308,"num_objects":5761,"num_object_clones":0,"num_object_copies":23044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6799,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3426}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6777","reported_seq":11440,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:59.740127+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:59.740127+0000","last_peered":"2024-09-15T21:07:59.740127+0000","last_clean":"2024-09-15T21:07:59.740127+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:59.740127+0000","last_undegraded":"2024-09-15T21:07:59.740127+0000","last_fullsized":"2024-09-15T21:07:59.740127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45735936,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6745","reported_seq":6785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.748566+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:59.748566+0000","last_peered":"2024-09-15T21:07:59.748566+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:59.748566+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46545542,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6780","reported_seq":6858,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.422945+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:07:57.422945+0000","last_peered":"2024-09-15T21:07:57.422945+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:07:57.422945+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6780,"log_dups_size":0,"ondisk_log_size":6780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46865030,"num_objects":5721,"num_object_clones":0,"num_object_copies":22884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3435}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6772","reported_seq":6829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.854603+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:59.854603+0000","last_peered":"2024-09-15T21:07:59.854603+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:59.854603+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6772,"log_dups_size":0,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6862","reported_seq":6945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:57.377612+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:07:57.377612+0000","last_peered":"2024-09-15T21:07:57.377612+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:07:57.377612+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6862,"log_dups_size":0,"ondisk_log_size":6862,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47666444,"num_objects":5819,"num_object_clones":0,"num_object_copies":23276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6860,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3538}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6877","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.517592+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:59.517592+0000","last_peered":"2024-09-15T21:07:59.517592+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:59.517592+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":746706734,"num_objects":91156,"num_object_clones":0,"num_object_copies":364624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91156,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103256,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1386110976,"data_stored":1386110976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109169,"ondisk_log_size":109169,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752984,"kb_used_data":364484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93618856,"statfs":{"total":96636764160,"available":95865708544,"internally_reserved":0,"allocated":373231616,"data_stored":373066627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741264,"kb_used_data":367156,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":93630576,"statfs":{"total":96636764160,"available":95877709824,"internally_reserved":0,"allocated":375967744,"data_stored":375802755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051633,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":530076,"kb_used_data":259660,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93841764,"statfs":{"total":96636764160,"available":96093966336,"internally_reserved":0,"allocated":265891840,"data_stored":265726851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765424,"kb_used_data":364504,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93606416,"statfs":{"total":96636764160,"available":95852969984,"internally_reserved":0,"allocated":373252096,"data_stored":373087107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372887552,"data_stored":372887552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265560064,"data_stored":265560064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":375115776,"data_stored":375115776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372547584,"data_stored":372547584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:05.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:05.723 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:05.724 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:05.756 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-15T21:08:04.017704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":744167214,"num_objects":90846,"num_object_clones":0,"num_object_copies":363384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103566,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109479,"ondisk_log_size":109479,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2810628,"kb_used_data":1350328,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374676732,"statfs":{"total":386547056640,"available":383668973568,"internally_reserved":0,"allocated":1382735872,"data_stored":1382075916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1495254578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,5,17,9,19,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23953408,"num_objects":-2924,"num_object_clones":0,"num_object_copies":-11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":712704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005777"},"pg_stats":[{"pgid":"1.f","version":"33'6736","reported_seq":6782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.619361+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:07:59.619361+0000","last_peered":"2024-09-15T21:07:59.619361+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:07:59.619361+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6736,"log_dups_size":0,"ondisk_log_size":6736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":5616,"num_object_clones":0,"num_object_copies":22464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6736,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6845","reported_seq":8145,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.513912+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:07:59.513912+0000","last_peered":"2024-09-15T21:07:59.513912+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:07:59.513912+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6845,"log_dups_size":0,"ondisk_log_size":6845,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46748940,"num_objects":5707,"num_object_clones":0,"num_object_copies":22828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1764,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6843,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3943}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6828","reported_seq":6908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.488391+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:02.488391+0000","last_peered":"2024-09-15T21:08:02.488391+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:02.488391+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45906590,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6827,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6810","reported_seq":6879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.527250+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:07:59.527250+0000","last_peered":"2024-09-15T21:07:59.527250+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:07:59.527250+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6810,"log_dups_size":0,"ondisk_log_size":6810,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46249228,"num_objects":5646,"num_object_clones":0,"num_object_copies":22584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6808,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3312}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6903","reported_seq":6959,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.847582+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:07:59.847582+0000","last_peered":"2024-09-15T21:07:59.847582+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:07:59.847582+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6903,"log_dups_size":0,"ondisk_log_size":6903,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47150348,"num_objects":5756,"num_object_clones":0,"num_object_copies":23024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6901,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3445}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6749","reported_seq":6792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.817742+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:07:59.817742+0000","last_peered":"2024-09-15T21:07:59.817742+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:07:59.817742+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6749,"log_dups_size":0,"ondisk_log_size":6749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46096384,"num_objects":5627,"num_object_clones":0,"num_object_copies":22508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6749,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3311}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'6986","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.399455+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:07:59.399455+0000","last_peered":"2024-09-15T21:07:59.399455+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:07:59.399455+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6986,"log_dups_size":0,"ondisk_log_size":6986,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47396108,"num_objects":5786,"num_object_clones":0,"num_object_copies":23144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3420}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'6978","reported_seq":7047,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.397305+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:07:59.397305+0000","last_peered":"2024-09-15T21:07:59.397305+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:07:59.397305+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47371532,"num_objects":5783,"num_object_clones":0,"num_object_copies":23132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6976,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6800","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.782919+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:07:59.782919+0000","last_peered":"2024-09-15T21:07:59.782919+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:07:59.782919+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6800,"log_dups_size":0,"ondisk_log_size":6800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46295576,"num_objects":5652,"num_object_clones":0,"num_object_copies":22608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6797,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3320}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6880","reported_seq":6961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.404317+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:02.404317+0000","last_peered":"2024-09-15T21:08:02.404317+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:02.404317+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544140,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6777","reported_seq":11440,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:07:59.740127+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:07:59.740127+0000","last_peered":"2024-09-15T21:07:59.740127+0000","last_clean":"2024-09-15T21:07:59.740127+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:07:59.740127+0000","last_undegraded":"2024-09-15T21:07:59.740127+0000","last_fullsized":"2024-09-15T21:07:59.740127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6777,"log_dups_size":0,"ondisk_log_size":6777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45735936,"num_objects":5583,"num_object_clones":0,"num_object_copies":22332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6745","reported_seq":6785,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.748566+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:07:59.748566+0000","last_peered":"2024-09-15T21:07:59.748566+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:07:59.748566+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46545542,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6744,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6848","reported_seq":6928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.484955+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:02.484955+0000","last_peered":"2024-09-15T21:08:02.484955+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:02.484955+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307974,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6772","reported_seq":6829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.854603+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:07:59.854603+0000","last_peered":"2024-09-15T21:07:59.854603+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:07:59.854603+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6772,"log_dups_size":0,"ondisk_log_size":6772,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6770,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3381}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6945","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.471054+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:02.471054+0000","last_peered":"2024-09-15T21:08:02.471054+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:02.471054+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46986508,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6877","reported_seq":6944,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:07:59.517592+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:07:59.517592+0000","last_peered":"2024-09-15T21:07:59.517592+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:07:59.517592+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46462220,"num_objects":5672,"num_object_clones":0,"num_object_copies":22688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6875,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3357}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":744167214,"num_objects":90846,"num_object_clones":0,"num_object_copies":363384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34205,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103566,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1380511744,"data_stored":1380511744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109479,"ondisk_log_size":109479,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752984,"kb_used_data":364484,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93618856,"statfs":{"total":96636764160,"available":95865708544,"internally_reserved":0,"allocated":373231616,"data_stored":373066627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,9,0,11],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758376,"kb_used_data":362072,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93613464,"statfs":{"total":96636764160,"available":95860187136,"internally_reserved":0,"allocated":370761728,"data_stored":370596739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533844,"kb_used_data":259268,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93837996,"statfs":{"total":96636764160,"available":96090107904,"internally_reserved":0,"allocated":265490432,"data_stored":265325443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":765424,"kb_used_data":364504,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93606416,"statfs":{"total":96636764160,"available":95852969984,"internally_reserved":0,"allocated":373252096,"data_stored":373087107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,3,3,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":372887552,"data_stored":372887552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265134080,"data_stored":265134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369942528,"data_stored":369942528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":372547584,"data_stored":372547584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:05.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:06.426 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:06.427 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:06.455 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T21:08:06.018372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":736204590,"num_objects":89874,"num_object_clones":0,"num_object_copies":359496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89874,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104538,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110451,"ondisk_log_size":110451,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805532,"kb_used_data":1341080,"kb_used_omap":14,"kb_used_meta":1464369,"kb_avail":374681828,"statfs":{"total":386547056640,"available":383674191872,"internally_reserved":0,"allocated":1373265920,"data_stored":1372605964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1499514418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,11,9,17,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29270016,"num_objects":-3573,"num_object_clones":0,"num_object_copies":-14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004245"},"pg_stats":[{"pgid":"1.f","version":"33'6831","reported_seq":6877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.045584+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:05.045584+0000","last_peered":"2024-09-15T21:08:05.045584+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:05.045584+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6831,"log_dups_size":0,"ondisk_log_size":6831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45228032,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6923","reported_seq":8316,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.461559+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:04.461559+0000","last_peered":"2024-09-15T21:08:04.461559+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:04.461559+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46109964,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6828","reported_seq":6908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.488391+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:02.488391+0000","last_peered":"2024-09-15T21:08:02.488391+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:02.488391+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45906590,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6827,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6884","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.452860+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:04.452860+0000","last_peered":"2024-09-15T21:08:04.452860+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:04.452860+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6884,"log_dups_size":0,"ondisk_log_size":6884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45643020,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6882,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6988","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.872039+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:04.872039+0000","last_peered":"2024-09-15T21:08:04.872039+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:04.872039+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46454028,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6853","reported_seq":6897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.009346+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:05.009346+0000","last_peered":"2024-09-15T21:08:05.009346+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:05.009346+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45244416,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7055","reported_seq":7125,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.516861+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:04.516861+0000","last_peered":"2024-09-15T21:08:04.516861+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:04.516861+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830860,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7055","reported_seq":7124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.459432+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:04.459432+0000","last_peered":"2024-09-15T21:08:04.459432+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:04.459432+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46740748,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6895","reported_seq":6955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.929424+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:04.929424+0000","last_peered":"2024-09-15T21:08:04.929424+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:04.929424+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45517336,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6892,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6880","reported_seq":6961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.404317+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:02.404317+0000","last_peered":"2024-09-15T21:08:02.404317+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:02.404317+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544140,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6850","reported_seq":11513,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:05.009280+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:05.009280+0000","last_peered":"2024-09-15T21:08:05.009280+0000","last_clean":"2024-09-15T21:08:05.009280+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:05.009280+0000","last_undegraded":"2024-09-15T21:08:05.009280+0000","last_fullsized":"2024-09-15T21:08:05.009280+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45137920,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6820","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.927789+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:04.927789+0000","last_peered":"2024-09-15T21:08:04.927789+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:04.927789+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45931142,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6848","reported_seq":6928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.484955+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:02.484955+0000","last_peered":"2024-09-15T21:08:02.484955+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:02.484955+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307974,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6844","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.967217+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:04.967217+0000","last_peered":"2024-09-15T21:08:04.967217+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:04.967217+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45774092,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6945","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.471054+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:02.471054+0000","last_peered":"2024-09-15T21:08:02.471054+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:02.471054+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46986508,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6952","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.448907+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:04.448907+0000","last_peered":"2024-09-15T21:08:04.448907+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:04.448907+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847820,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":736204590,"num_objects":89874,"num_object_clones":0,"num_object_copies":359496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89874,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104538,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369935872,"data_stored":1369935872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110451,"ondisk_log_size":110451,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752000,"kb_used_data":359344,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93619840,"statfs":{"total":96636764160,"available":95866716160,"internally_reserved":0,"allocated":367968256,"data_stored":367803267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758376,"kb_used_data":362072,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93613464,"statfs":{"total":96636764160,"available":95860187136,"internally_reserved":0,"allocated":370761728,"data_stored":370596739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533844,"kb_used_data":259268,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93837996,"statfs":{"total":96636764160,"available":96090107904,"internally_reserved":0,"allocated":265490432,"data_stored":265325443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761312,"kb_used_data":360396,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93610528,"statfs":{"total":96636764160,"available":95857180672,"internally_reserved":0,"allocated":369045504,"data_stored":368880515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367689728,"data_stored":367689728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265134080,"data_stored":265134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369942528,"data_stored":369942528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367169536,"data_stored":367169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:06.456 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:06.456 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6831", 'reported_seq': 6877, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:05.045584+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:05.045584+0000', 'last_peered': '2024-09-15T21:08:05.045584+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:05.045584+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6831, 'log_dups_size': 0, 'ondisk_log_size': 6831, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45228032, 'num_objects': 5521, 'num_object_clones': 0, 'num_object_copies': 22084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6831, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3245}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6923", 'reported_seq': 8316, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.461559+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:04.461559+0000', 'last_peered': '2024-09-15T21:08:04.461559+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:04.461559+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6923, 'log_dups_size': 0, 'ondisk_log_size': 6923, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46109964, 'num_objects': 5629, 'num_object_clones': 0, 'num_object_copies': 22516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1718, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6921, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 664, 'num_bytes_recovered': 5423104, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6828", 'reported_seq': 6908, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:02.488391+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:02.488391+0000', 'last_peered': '2024-09-15T21:08:02.488391+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:02.488391+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6828, 'log_dups_size': 0, 'ondisk_log_size': 6828, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45906590, 'num_objects': 5605, 'num_object_clones': 0, 'num_object_copies': 22420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5605, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6827, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3347}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6884", 'reported_seq': 6953, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.452860+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:04.452860+0000', 'last_peered': '2024-09-15T21:08:04.452860+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:04.452860+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6884, 'log_dups_size': 0, 'ondisk_log_size': 6884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45643020, 'num_objects': 5572, 'num_object_clones': 0, 'num_object_copies': 22288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6882, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'6988", 'reported_seq': 7044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.872039+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:04.872039+0000', 'last_peered': '2024-09-15T21:08:04.872039+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:04.872039+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6988, 'log_dups_size': 0, 'ondisk_log_size': 6988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46454028, 'num_objects': 5671, 'num_object_clones': 0, 'num_object_copies': 22684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5671, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6986, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3360}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6853", 'reported_seq': 6897, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:05.009346+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:05.009346+0000', 'last_peered': '2024-09-15T21:08:05.009346+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:05.009346+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6853, 'log_dups_size': 0, 'ondisk_log_size': 6853, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45244416, 'num_objects': 5523, 'num_object_clones': 0, 'num_object_copies': 22092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6853, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3207}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.457 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7055", 'reported_seq': 7125, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.516861+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:04.516861+0000', 'last_peered': '2024-09-15T21:08:04.516861+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:04.516861+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46830860, 'num_objects': 5717, 'num_object_clones': 0, 'num_object_copies': 22868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5717, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7053, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7055", 'reported_seq': 7124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.459432+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:04.459432+0000', 'last_peered': '2024-09-15T21:08:04.459432+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:04.459432+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7055, 'log_dups_size': 0, 'ondisk_log_size': 7055, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46740748, 'num_objects': 5706, 'num_object_clones': 0, 'num_object_copies': 22824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5706, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7053, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3232}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6895", 'reported_seq': 6955, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.929424+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:04.929424+0000', 'last_peered': '2024-09-15T21:08:04.929424+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:04.929424+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6895, 'log_dups_size': 0, 'ondisk_log_size': 6895, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45517336, 'num_objects': 5557, 'num_object_clones': 0, 'num_object_copies': 22228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5557, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6892, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3225}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6880", 'reported_seq': 6961, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:02.404317+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:02.404317+0000', 'last_peered': '2024-09-15T21:08:02.404317+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:02.404317+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6880, 'log_dups_size': 0, 'ondisk_log_size': 6880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46544140, 'num_objects': 5682, 'num_object_clones': 0, 'num_object_copies': 22728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5682, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6878, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3347}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6820", 'reported_seq': 6860, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.927789+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:04.927789+0000', 'last_peered': '2024-09-15T21:08:04.927789+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:04.927789+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6820, 'log_dups_size': 0, 'ondisk_log_size': 6820, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45931142, 'num_objects': 5607, 'num_object_clones': 0, 'num_object_copies': 22428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5607, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6819, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.458 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6848", 'reported_seq': 6928, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:02.484955+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:02.484955+0000', 'last_peered': '2024-09-15T21:08:02.484955+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:02.484955+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6848, 'log_dups_size': 0, 'ondisk_log_size': 6848, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46307974, 'num_objects': 5653, 'num_object_clones': 0, 'num_object_copies': 22612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5653, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6847, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3367}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6844", 'reported_seq': 6899, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.967217+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:04.967217+0000', 'last_peered': '2024-09-15T21:08:04.967217+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:04.967217+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6844, 'log_dups_size': 0, 'ondisk_log_size': 6844, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45774092, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6842, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'6945", 'reported_seq': 7028, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:02.471054+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:02.471054+0000', 'last_peered': '2024-09-15T21:08:02.471054+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:02.471054+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6945, 'log_dups_size': 0, 'ondisk_log_size': 6945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46986508, 'num_objects': 5736, 'num_object_clones': 0, 'num_object_copies': 22944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5736, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6943, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3455}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:06.459 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'6952", 'reported_seq': 7021, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:04.448907+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:04.448907+0000', 'last_peered': '2024-09-15T21:08:04.448907+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:04.448907+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6952, 'log_dups_size': 0, 'ondisk_log_size': 6952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45847820, 'num_objects': 5597, 'num_object_clones': 0, 'num_object_copies': 22388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6950, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:06.459 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:07.209 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:07.209 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34159/359496 objects degraded (9.502%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":352,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":89874,"data_bytes":736204590,"bytes_used":2872864768,"bytes_avail":383674191872,"bytes_total":386547056640,"degraded_objects":34159,"degraded_total":359496,"degraded_ratio":0.095019137904176959,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":90762,"recovering_keys_per_sec":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":297},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:07.252 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:07.253 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:07.868 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:07.869 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:07.897 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-15T21:08:06.018372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":736204590,"num_objects":89874,"num_object_clones":0,"num_object_copies":359496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89874,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104538,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110451,"ondisk_log_size":110451,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805532,"kb_used_data":1341080,"kb_used_omap":14,"kb_used_meta":1464369,"kb_avail":374681828,"statfs":{"total":386547056640,"available":383674191872,"internally_reserved":0,"allocated":1373265920,"data_stored":1372605964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1499514418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,11,9,17,12,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-29270016,"num_objects":-3573,"num_object_clones":0,"num_object_copies":-14292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3573,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004245"},"pg_stats":[{"pgid":"1.f","version":"33'6831","reported_seq":6877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.045584+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:05.045584+0000","last_peered":"2024-09-15T21:08:05.045584+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:05.045584+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6831,"log_dups_size":0,"ondisk_log_size":6831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45228032,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6923","reported_seq":8316,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.461559+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:04.461559+0000","last_peered":"2024-09-15T21:08:04.461559+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:04.461559+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6923,"log_dups_size":0,"ondisk_log_size":6923,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46109964,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1718,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6921,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6828","reported_seq":6908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.488391+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:02.488391+0000","last_peered":"2024-09-15T21:08:02.488391+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:02.488391+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6828,"log_dups_size":0,"ondisk_log_size":6828,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45906590,"num_objects":5605,"num_object_clones":0,"num_object_copies":22420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6827,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6884","reported_seq":6953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.452860+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:04.452860+0000","last_peered":"2024-09-15T21:08:04.452860+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:04.452860+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6884,"log_dups_size":0,"ondisk_log_size":6884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45643020,"num_objects":5572,"num_object_clones":0,"num_object_copies":22288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6882,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6988","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.872039+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:04.872039+0000","last_peered":"2024-09-15T21:08:04.872039+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:04.872039+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46454028,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6853","reported_seq":6897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.009346+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:05.009346+0000","last_peered":"2024-09-15T21:08:05.009346+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:05.009346+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45244416,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7055","reported_seq":7125,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.516861+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:04.516861+0000","last_peered":"2024-09-15T21:08:04.516861+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:04.516861+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46830860,"num_objects":5717,"num_object_clones":0,"num_object_copies":22868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7055","reported_seq":7124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.459432+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:04.459432+0000","last_peered":"2024-09-15T21:08:04.459432+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:04.459432+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7055,"log_dups_size":0,"ondisk_log_size":7055,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46740748,"num_objects":5706,"num_object_clones":0,"num_object_copies":22824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7053,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3232}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6895","reported_seq":6955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.929424+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:04.929424+0000","last_peered":"2024-09-15T21:08:04.929424+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:04.929424+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45517336,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6892,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6880","reported_seq":6961,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.404317+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:02.404317+0000","last_peered":"2024-09-15T21:08:02.404317+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:02.404317+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46544140,"num_objects":5682,"num_object_clones":0,"num_object_copies":22728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6878,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6850","reported_seq":11513,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:05.009280+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:05.009280+0000","last_peered":"2024-09-15T21:08:05.009280+0000","last_clean":"2024-09-15T21:08:05.009280+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:05.009280+0000","last_undegraded":"2024-09-15T21:08:05.009280+0000","last_fullsized":"2024-09-15T21:08:05.009280+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45137920,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6820","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.927789+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:04.927789+0000","last_peered":"2024-09-15T21:08:04.927789+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:04.927789+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45931142,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6848","reported_seq":6928,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.484955+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:02.484955+0000","last_peered":"2024-09-15T21:08:02.484955+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:02.484955+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6848,"log_dups_size":0,"ondisk_log_size":6848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46307974,"num_objects":5653,"num_object_clones":0,"num_object_copies":22612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3367}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6844","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.967217+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:04.967217+0000","last_peered":"2024-09-15T21:08:04.967217+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:04.967217+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45774092,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'6945","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:02.471054+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:02.471054+0000","last_peered":"2024-09-15T21:08:02.471054+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:02.471054+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6945,"log_dups_size":0,"ondisk_log_size":6945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46986508,"num_objects":5736,"num_object_clones":0,"num_object_copies":22944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3455}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'6952","reported_seq":7021,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.448907+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:04.448907+0000","last_peered":"2024-09-15T21:08:04.448907+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:04.448907+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45847820,"num_objects":5597,"num_object_clones":0,"num_object_copies":22388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6950,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":736204590,"num_objects":89874,"num_object_clones":0,"num_object_copies":359496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34159,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89874,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":104538,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":686,"num_bytes_recovered":5423104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1369935872,"data_stored":1369935872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110451,"ondisk_log_size":110451,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752000,"kb_used_data":359344,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93619840,"statfs":{"total":96636764160,"available":95866716160,"internally_reserved":0,"allocated":367968256,"data_stored":367803267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758376,"kb_used_data":362072,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93613464,"statfs":{"total":96636764160,"available":95860187136,"internally_reserved":0,"allocated":370761728,"data_stored":370596739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051634,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":533844,"kb_used_data":259268,"kb_used_omap":4,"kb_used_meta":274555,"kb_avail":93837996,"statfs":{"total":96636764160,"available":96090107904,"internally_reserved":0,"allocated":265490432,"data_stored":265325443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":281145039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,5,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":761312,"kb_used_data":360396,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93610528,"statfs":{"total":96636764160,"available":95857180672,"internally_reserved":0,"allocated":369045504,"data_stored":368880515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,5,3,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":367689728,"data_stored":367689728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":265134080,"data_stored":265134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":369942528,"data_stored":369942528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367169536,"data_stored":367169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:10.899 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:11.586 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:11.587 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:11.612 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-15T21:08:10.020032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":730388270,"num_objects":89164,"num_object_clones":0,"num_object_copies":356656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89164,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105248,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111161,"ondisk_log_size":111161,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2782336,"kb_used_data":1330368,"kb_used_omap":14,"kb_used_meta":1451889,"kb_avail":374705024,"statfs":{"total":386547056640,"available":383697944576,"internally_reserved":0,"allocated":1362296832,"data_stored":1361636876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1486734898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,9,6,17,8,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24313856,"num_objects":-2968,"num_object_clones":0,"num_object_copies":-11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-135,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2968,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1105920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004620"},"pg_stats":[{"pgid":"1.f","version":"33'6831","reported_seq":6877,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.045584+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:05.045584+0000","last_peered":"2024-09-15T21:08:05.045584+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:05.045584+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6831,"log_dups_size":0,"ondisk_log_size":6831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45228032,"num_objects":5521,"num_object_clones":0,"num_object_copies":22084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6831,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6997","reported_seq":8477,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.630966+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:09.630966+0000","last_peered":"2024-09-15T21:08:09.630966+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:09.630966+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503756,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6905","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.364914+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:07.364914+0000","last_peered":"2024-09-15T21:08:07.364914+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:07.364914+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45275806,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6904,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6968","reported_seq":7037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.632227+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:09.632227+0000","last_peered":"2024-09-15T21:08:09.632227+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:09.632227+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44954892,"num_objects":5488,"num_object_clones":0,"num_object_copies":21952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6966,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'6988","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.872039+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:04.872039+0000","last_peered":"2024-09-15T21:08:04.872039+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:04.872039+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46454028,"num_objects":5671,"num_object_clones":0,"num_object_copies":22684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6986,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3360}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6853","reported_seq":6897,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:05.009346+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:05.009346+0000","last_peered":"2024-09-15T21:08:05.009346+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:05.009346+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45244416,"num_objects":5523,"num_object_clones":0,"num_object_copies":22092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6853,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3207}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7133","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.650921+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:09.650921+0000","last_peered":"2024-09-15T21:08:09.650921+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:09.650921+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7132","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.593937+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:09.593937+0000","last_peered":"2024-09-15T21:08:09.593937+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:09.593937+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46109964,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6895","reported_seq":6955,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.929424+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:04.929424+0000","last_peered":"2024-09-15T21:08:04.929424+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:04.929424+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45517336,"num_objects":5557,"num_object_clones":0,"num_object_copies":22228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6892,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3225}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6963","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.279502+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:07.279502+0000","last_peered":"2024-09-15T21:08:07.279502+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:07.279502+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45864204,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6850","reported_seq":11513,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:05.009280+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:05.009280+0000","last_peered":"2024-09-15T21:08:05.009280+0000","last_clean":"2024-09-15T21:08:05.009280+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:05.009280+0000","last_undegraded":"2024-09-15T21:08:05.009280+0000","last_fullsized":"2024-09-15T21:08:05.009280+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45137920,"num_objects":5510,"num_object_clones":0,"num_object_copies":22040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6820","reported_seq":6860,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.927789+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:04.927789+0000","last_peered":"2024-09-15T21:08:04.927789+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:04.927789+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6820,"log_dups_size":0,"ondisk_log_size":6820,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45931142,"num_objects":5607,"num_object_clones":0,"num_object_copies":22428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3351}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6936","reported_seq":7016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.390803+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:07.390803+0000","last_peered":"2024-09-15T21:08:07.390803+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:07.390803+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587078,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6935,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6844","reported_seq":6899,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:04.967217+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:04.967217+0000","last_peered":"2024-09-15T21:08:04.967217+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:04.967217+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45774092,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6842,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3309}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7021","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.487803+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:07.487803+0000","last_peered":"2024-09-15T21:08:07.487803+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:07.487803+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7025","reported_seq":7094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.656128+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:09.656128+0000","last_peered":"2024-09-15T21:08:09.656128+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:09.656128+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249804,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":730388270,"num_objects":89164,"num_object_clones":0,"num_object_copies":356656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89164,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105248,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359167488,"data_stored":1359167488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111161,"ondisk_log_size":111161,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752000,"kb_used_data":359344,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93619840,"statfs":{"total":96636764160,"available":95866716160,"internally_reserved":0,"allocated":367968256,"data_stored":367803267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,6,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757316,"kb_used_data":356852,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93614524,"statfs":{"total":96636764160,"available":95861272576,"internally_reserved":0,"allocated":365416448,"data_stored":365251459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525080,"kb_used_data":258824,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93846760,"statfs":{"total":96636764160,"available":96099082240,"internally_reserved":0,"allocated":265035776,"data_stored":264870787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747940,"kb_used_data":355348,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93623900,"statfs":{"total":96636764160,"available":95870873600,"internally_reserved":0,"allocated":363876352,"data_stored":363711363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362475520,"data_stored":362475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264736768,"data_stored":264736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364785664,"data_stored":364785664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":367169536,"data_stored":367169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:11.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:12.219 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:12.220 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:12.244 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-15T21:08:12.020706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725833518,"num_objects":88608,"num_object_clones":0,"num_object_copies":354432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105804,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111717,"ondisk_log_size":111717,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785808,"kb_used_data":1325520,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374701552,"statfs":{"total":386547056640,"available":383694389248,"internally_reserved":0,"allocated":1357332480,"data_stored":1356672524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1495254578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,2,4,6,5,12,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20873216,"num_objects":-2548,"num_object_clones":0,"num_object_copies":-10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004389"},"pg_stats":[{"pgid":"1.f","version":"33'6906","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.129341+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:10.129341+0000","last_peered":"2024-09-15T21:08:10.129341+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:10.129341+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44613632,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6997","reported_seq":8477,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.630966+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:09.630966+0000","last_peered":"2024-09-15T21:08:09.630966+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:09.630966+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503756,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6905","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.364914+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:07.364914+0000","last_peered":"2024-09-15T21:08:07.364914+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:07.364914+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45275806,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6904,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6968","reported_seq":7037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.632227+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:09.632227+0000","last_peered":"2024-09-15T21:08:09.632227+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:09.632227+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44954892,"num_objects":5488,"num_object_clones":0,"num_object_copies":21952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6966,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7071","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.034351+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:10.034351+0000","last_peered":"2024-09-15T21:08:10.034351+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:10.034351+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7071,"log_dups_size":0,"ondisk_log_size":7071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45774092,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7069,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6927","reported_seq":6972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.977404+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:09.977404+0000","last_peered":"2024-09-15T21:08:09.977404+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:09.977404+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44638208,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7133","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.650921+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:09.650921+0000","last_peered":"2024-09-15T21:08:09.650921+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:09.650921+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7132","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.593937+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:09.593937+0000","last_peered":"2024-09-15T21:08:09.593937+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:09.593937+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46109964,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6978","reported_seq":7038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.080431+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:10.080431+0000","last_peered":"2024-09-15T21:08:10.080431+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:10.080431+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44837400,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6963","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.279502+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:07.279502+0000","last_peered":"2024-09-15T21:08:07.279502+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:07.279502+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45864204,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6927","reported_seq":11589,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:10.037304+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:10.037304+0000","last_peered":"2024-09-15T21:08:10.037304+0000","last_clean":"2024-09-15T21:08:10.037304+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:10.037304+0000","last_undegraded":"2024-09-15T21:08:10.037304+0000","last_fullsized":"2024-09-15T21:08:10.037304+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44507136,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6910","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.901435+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:09.901435+0000","last_peered":"2024-09-15T21:08:09.901435+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:09.901435+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45193862,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6936","reported_seq":7016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.390803+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:07.390803+0000","last_peered":"2024-09-15T21:08:07.390803+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:07.390803+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587078,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6935,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6918","reported_seq":6975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.062729+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:10.062729+0000","last_peered":"2024-09-15T21:08:10.062729+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:10.062729+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7021","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.487803+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:07.487803+0000","last_peered":"2024-09-15T21:08:07.487803+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:07.487803+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7025","reported_seq":7094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.656128+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:09.656128+0000","last_peered":"2024-09-15T21:08:09.656128+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:09.656128+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249804,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725833518,"num_objects":88608,"num_object_clones":0,"num_object_copies":354432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105804,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1353936896,"data_stored":1353936896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111717,"ondisk_log_size":111717,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755472,"kb_used_data":354496,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93616368,"statfs":{"total":96636764160,"available":95863160832,"internally_reserved":0,"allocated":363003904,"data_stored":362838915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,0,3,0,0,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757316,"kb_used_data":356852,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93614524,"statfs":{"total":96636764160,"available":95861272576,"internally_reserved":0,"allocated":365416448,"data_stored":365251459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525080,"kb_used_data":258824,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93846760,"statfs":{"total":96636764160,"available":96099082240,"internally_reserved":0,"allocated":265035776,"data_stored":264870787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747940,"kb_used_data":355348,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93623900,"statfs":{"total":96636764160,"available":95870873600,"internally_reserved":0,"allocated":363876352,"data_stored":363711363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362475520,"data_stored":362475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264736768,"data_stored":264736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364785664,"data_stored":364785664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":361938944,"data_stored":361938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:12.245 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:12.245 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6906", 'reported_seq': 6950, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:10.129341+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:10.129341+0000', 'last_peered': '2024-09-15T21:08:10.129341+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:10.129341+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6906, 'log_dups_size': 0, 'ondisk_log_size': 6906, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44613632, 'num_objects': 5446, 'num_object_clones': 0, 'num_object_copies': 21784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6906, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3170}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'6997", 'reported_seq': 8477, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.630966+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:09.630966+0000', 'last_peered': '2024-09-15T21:08:09.630966+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:09.630966+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6997, 'log_dups_size': 0, 'ondisk_log_size': 6997, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45503756, 'num_objects': 5555, 'num_object_clones': 0, 'num_object_copies': 22220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1674, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6995, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 708, 'num_bytes_recovered': 5783552, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3881}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6905", 'reported_seq': 6985, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:07.364914+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:07.364914+0000', 'last_peered': '2024-09-15T21:08:07.364914+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:07.364914+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6905, 'log_dups_size': 0, 'ondisk_log_size': 6905, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45275806, 'num_objects': 5528, 'num_object_clones': 0, 'num_object_copies': 22112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5528, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6904, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3270}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'6968", 'reported_seq': 7037, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.632227+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:09.632227+0000', 'last_peered': '2024-09-15T21:08:09.632227+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:09.632227+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6968, 'log_dups_size': 0, 'ondisk_log_size': 6968, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44954892, 'num_objects': 5488, 'num_object_clones': 0, 'num_object_copies': 21952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6966, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7071", 'reported_seq': 7127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:10.034351+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:10.034351+0000', 'last_peered': '2024-09-15T21:08:10.034351+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:10.034351+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7071, 'log_dups_size': 0, 'ondisk_log_size': 7071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45774092, 'num_objects': 5588, 'num_object_clones': 0, 'num_object_copies': 22352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7069, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3277}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.246 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6927", 'reported_seq': 6972, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.977404+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:09.977404+0000', 'last_peered': '2024-09-15T21:08:09.977404+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:09.977404+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6927, 'log_dups_size': 0, 'ondisk_log_size': 6927, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44638208, 'num_objects': 5449, 'num_object_clones': 0, 'num_object_copies': 21796, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5449, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6927, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3133}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7133", 'reported_seq': 7203, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.650921+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:09.650921+0000', 'last_peered': '2024-09-15T21:08:09.650921+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:09.650921+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7133, 'log_dups_size': 0, 'ondisk_log_size': 7133, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46191884, 'num_objects': 5639, 'num_object_clones': 0, 'num_object_copies': 22556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7131, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3273}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7132", 'reported_seq': 7201, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.593937+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:09.593937+0000', 'last_peered': '2024-09-15T21:08:09.593937+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:09.593937+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7132, 'log_dups_size': 0, 'ondisk_log_size': 7132, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46109964, 'num_objects': 5629, 'num_object_clones': 0, 'num_object_copies': 22516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7130, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'6978", 'reported_seq': 7038, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:10.080431+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:10.080431+0000', 'last_peered': '2024-09-15T21:08:10.080431+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:10.080431+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6978, 'log_dups_size': 0, 'ondisk_log_size': 6978, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44837400, 'num_objects': 5474, 'num_object_clones': 0, 'num_object_copies': 21896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5474, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6975, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3142}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'6963", 'reported_seq': 7044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:07.279502+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:07.279502+0000', 'last_peered': '2024-09-15T21:08:07.279502+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:07.279502+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6963, 'log_dups_size': 0, 'ondisk_log_size': 6963, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45864204, 'num_objects': 5599, 'num_object_clones': 0, 'num_object_copies': 22396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5599, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6961, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3264}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:12.247 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6910", 'reported_seq': 6950, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.901435+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:09.901435+0000', 'last_peered': '2024-09-15T21:08:09.901435+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:09.901435+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6910, 'log_dups_size': 0, 'ondisk_log_size': 6910, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45193862, 'num_objects': 5517, 'num_object_clones': 0, 'num_object_copies': 22068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6909, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3261}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'6936", 'reported_seq': 7016, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:07.390803+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:07.390803+0000', 'last_peered': '2024-09-15T21:08:07.390803+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:07.390803+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6936, 'log_dups_size': 0, 'ondisk_log_size': 6936, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45587078, 'num_objects': 5565, 'num_object_clones': 0, 'num_object_copies': 22260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5565, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6935, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6918", 'reported_seq': 6975, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:10.062729+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:10.062729+0000', 'last_peered': '2024-09-15T21:08:10.062729+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:10.062729+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6918, 'log_dups_size': 0, 'ondisk_log_size': 6918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45167884, 'num_objects': 5514, 'num_object_clones': 0, 'num_object_copies': 22056, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5514, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6916, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7021", 'reported_seq': 7104, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:07.487803+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:07.487803+0000', 'last_peered': '2024-09-15T21:08:07.487803+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:07.487803+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7021, 'log_dups_size': 0, 'ondisk_log_size': 7021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46363916, 'num_objects': 5660, 'num_object_clones': 0, 'num_object_copies': 22640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5660, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7019, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3379}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7025", 'reported_seq': 7094, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:09.656128+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:09.656128+0000', 'last_peered': '2024-09-15T21:08:09.656128+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:09.656128+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7025, 'log_dups_size': 0, 'ondisk_log_size': 7025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45249804, 'num_objects': 5524, 'num_object_clones': 0, 'num_object_copies': 22096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7023, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3209}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:12.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:13.009 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:13.009 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34115/354432 objects degraded (9.625%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":358,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":88608,"data_bytes":725833518,"bytes_used":2852667392,"bytes_avail":383694389248,"bytes_total":386547056640,"degraded_objects":34115,"degraded_total":354432,"degraded_ratio":0.09625259570241966,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":61417,"recovering_keys_per_sec":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:13.029 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:13.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:13.649 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:13.650 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:13.669 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-15T21:08:12.020706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":725833518,"num_objects":88608,"num_object_clones":0,"num_object_copies":354432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105804,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111717,"ondisk_log_size":111717,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2785808,"kb_used_data":1325520,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374701552,"statfs":{"total":386547056640,"available":383694389248,"internally_reserved":0,"allocated":1357332480,"data_stored":1356672524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1495254578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[14,2,4,6,5,12,7,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20873216,"num_objects":-2548,"num_object_clones":0,"num_object_copies":-10192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2548,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004389"},"pg_stats":[{"pgid":"1.f","version":"33'6906","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.129341+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:10.129341+0000","last_peered":"2024-09-15T21:08:10.129341+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:10.129341+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6906,"log_dups_size":0,"ondisk_log_size":6906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44613632,"num_objects":5446,"num_object_clones":0,"num_object_copies":21784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6906,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3170}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'6997","reported_seq":8477,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.630966+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:09.630966+0000","last_peered":"2024-09-15T21:08:09.630966+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:09.630966+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45503756,"num_objects":5555,"num_object_clones":0,"num_object_copies":22220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6995,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":708,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3881}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6905","reported_seq":6985,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.364914+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:07.364914+0000","last_peered":"2024-09-15T21:08:07.364914+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:07.364914+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6905,"log_dups_size":0,"ondisk_log_size":6905,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45275806,"num_objects":5528,"num_object_clones":0,"num_object_copies":22112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5528,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6904,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3270}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'6968","reported_seq":7037,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.632227+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:09.632227+0000","last_peered":"2024-09-15T21:08:09.632227+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:09.632227+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6968,"log_dups_size":0,"ondisk_log_size":6968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44954892,"num_objects":5488,"num_object_clones":0,"num_object_copies":21952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6966,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7071","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.034351+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:10.034351+0000","last_peered":"2024-09-15T21:08:10.034351+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:10.034351+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7071,"log_dups_size":0,"ondisk_log_size":7071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45774092,"num_objects":5588,"num_object_clones":0,"num_object_copies":22352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7069,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3277}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6927","reported_seq":6972,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.977404+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:09.977404+0000","last_peered":"2024-09-15T21:08:09.977404+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:09.977404+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44638208,"num_objects":5449,"num_object_clones":0,"num_object_copies":21796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6927,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3133}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7133","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.650921+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:09.650921+0000","last_peered":"2024-09-15T21:08:09.650921+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:09.650921+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46191884,"num_objects":5639,"num_object_clones":0,"num_object_copies":22556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7131,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3273}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7132","reported_seq":7201,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.593937+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:09.593937+0000","last_peered":"2024-09-15T21:08:09.593937+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:09.593937+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7132,"log_dups_size":0,"ondisk_log_size":7132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46109964,"num_objects":5629,"num_object_clones":0,"num_object_copies":22516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7130,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'6978","reported_seq":7038,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.080431+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:10.080431+0000","last_peered":"2024-09-15T21:08:10.080431+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:10.080431+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44837400,"num_objects":5474,"num_object_clones":0,"num_object_copies":21896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6975,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3142}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'6963","reported_seq":7044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.279502+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:07.279502+0000","last_peered":"2024-09-15T21:08:07.279502+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:07.279502+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6963,"log_dups_size":0,"ondisk_log_size":6963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45864204,"num_objects":5599,"num_object_clones":0,"num_object_copies":22396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6961,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3264}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6927","reported_seq":11589,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:10.037304+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:10.037304+0000","last_peered":"2024-09-15T21:08:10.037304+0000","last_clean":"2024-09-15T21:08:10.037304+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:10.037304+0000","last_undegraded":"2024-09-15T21:08:10.037304+0000","last_fullsized":"2024-09-15T21:08:10.037304+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6927,"log_dups_size":0,"ondisk_log_size":6927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44507136,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6910","reported_seq":6950,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.901435+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:09.901435+0000","last_peered":"2024-09-15T21:08:09.901435+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:09.901435+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6910,"log_dups_size":0,"ondisk_log_size":6910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45193862,"num_objects":5517,"num_object_clones":0,"num_object_copies":22068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6909,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3261}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'6936","reported_seq":7016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.390803+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:07.390803+0000","last_peered":"2024-09-15T21:08:07.390803+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:07.390803+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6936,"log_dups_size":0,"ondisk_log_size":6936,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45587078,"num_objects":5565,"num_object_clones":0,"num_object_copies":22260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6935,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6918","reported_seq":6975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:10.062729+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:10.062729+0000","last_peered":"2024-09-15T21:08:10.062729+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:10.062729+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45167884,"num_objects":5514,"num_object_clones":0,"num_object_copies":22056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7021","reported_seq":7104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:07.487803+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:07.487803+0000","last_peered":"2024-09-15T21:08:07.487803+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:07.487803+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46363916,"num_objects":5660,"num_object_clones":0,"num_object_copies":22640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7019,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3379}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7025","reported_seq":7094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:09.656128+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:09.656128+0000","last_peered":"2024-09-15T21:08:09.656128+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:09.656128+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7025,"log_dups_size":0,"ondisk_log_size":7025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45249804,"num_objects":5524,"num_object_clones":0,"num_object_copies":22096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3209}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":725833518,"num_objects":88608,"num_object_clones":0,"num_object_copies":354432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":105804,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":730,"num_bytes_recovered":5783552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1353936896,"data_stored":1353936896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111717,"ondisk_log_size":111717,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":755472,"kb_used_data":354496,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93616368,"statfs":{"total":96636764160,"available":95863160832,"internally_reserved":0,"allocated":363003904,"data_stored":362838915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,0,3,0,0,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757316,"kb_used_data":356852,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93614524,"statfs":{"total":96636764160,"available":95861272576,"internally_reserved":0,"allocated":365416448,"data_stored":365251459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051635,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525080,"kb_used_data":258824,"kb_used_omap":4,"kb_used_meta":266235,"kb_avail":93846760,"statfs":{"total":96636764160,"available":96099082240,"internally_reserved":0,"allocated":265035776,"data_stored":264870787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":272625359},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747940,"kb_used_data":355348,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93623900,"statfs":{"total":96636764160,"available":95870873600,"internally_reserved":0,"allocated":363876352,"data_stored":363711363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":362475520,"data_stored":362475520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264736768,"data_stored":264736768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":364785664,"data_stored":364785664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":361938944,"data_stored":361938944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:16.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:17.320 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:17.321 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:17.349 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T21:08:16.022316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715306798,"num_objects":87323,"num_object_clones":0,"num_object_copies":349292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107089,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113002,"ondisk_log_size":113002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770300,"kb_used_data":1310000,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374717060,"statfs":{"total":386547056640,"available":383710269440,"internally_reserved":0,"allocated":1341440000,"data_stored":1340780044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1495254578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,8,10,9,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28860416,"num_objects":-3523,"num_object_clones":0,"num_object_copies":-14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004611"},"pg_stats":[{"pgid":"1.f","version":"33'6985","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.166766+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:15.166766+0000","last_peered":"2024-09-15T21:08:15.166766+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:15.166766+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6985,"log_dups_size":0,"ondisk_log_size":6985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43966464,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7084","reported_seq":8661,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.794917+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:14.794917+0000","last_peered":"2024-09-15T21:08:14.794917+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:14.794917+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44791052,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6989","reported_seq":7069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.434127+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:12.434127+0000","last_peered":"2024-09-15T21:08:12.434127+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:12.434127+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587678,"num_objects":5444,"num_object_clones":0,"num_object_copies":21776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6988,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3186}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7039","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.752949+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:14.752949+0000","last_peered":"2024-09-15T21:08:14.752949+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:14.752949+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373260,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7150","reported_seq":7206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.891002+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:14.891002+0000","last_peered":"2024-09-15T21:08:14.891002+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:14.891002+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45126924,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6996","reported_seq":7041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122325+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:15.122325+0000","last_peered":"2024-09-15T21:08:15.122325+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:15.122325+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44072960,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7230","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.677659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:14.677659+0000","last_peered":"2024-09-15T21:08:14.677659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:14.677659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397260,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7211","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.530990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:14.530990+0000","last_peered":"2024-09-15T21:08:14.530990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:14.530990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45462796,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7058","reported_seq":7118,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122718+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:15.122718+0000","last_peered":"2024-09-15T21:08:15.122718+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:15.122718+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44182040,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7051","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.424751+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:12.424751+0000","last_peered":"2024-09-15T21:08:12.424751+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:12.424751+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7051,"log_dups_size":0,"ondisk_log_size":7051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143308,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3176}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6997","reported_seq":11660,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:15.075659+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:15.075659+0000","last_peered":"2024-09-15T21:08:15.075659+0000","last_clean":"2024-09-15T21:08:15.075659+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:15.075659+0000","last_undegraded":"2024-09-15T21:08:15.075659+0000","last_fullsized":"2024-09-15T21:08:15.075659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1108,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6988","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.164659+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:15.164659+0000","last_peered":"2024-09-15T21:08:15.164659+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:15.164659+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44554886,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6987,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7014","reported_seq":7094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.297091+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:12.297091+0000","last_peered":"2024-09-15T21:08:12.297091+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:12.297091+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7014,"log_dups_size":0,"ondisk_log_size":7014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948102,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6999","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.115113+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:15.115113+0000","last_peered":"2024-09-15T21:08:15.115113+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:15.115113+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44504332,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7101","reported_seq":7184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.281133+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:12.281133+0000","last_peered":"2024-09-15T21:08:12.281133+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:12.281133+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7110","reported_seq":7179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.741608+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:14.741608+0000","last_peered":"2024-09-15T21:08:14.741608+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:14.741608+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553484,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715306798,"num_objects":87323,"num_object_clones":0,"num_object_copies":349292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107089,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1338249216,"data_stored":1338249216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113002,"ondisk_log_size":113002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754556,"kb_used_data":349416,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93617284,"statfs":{"total":96636764160,"available":95864098816,"internally_reserved":0,"allocated":357801984,"data_stored":357636995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739784,"kb_used_data":351792,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93632056,"statfs":{"total":96636764160,"available":95879225344,"internally_reserved":0,"allocated":360235008,"data_stored":360070019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529004,"kb_used_data":258588,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93842836,"statfs":{"total":96636764160,"available":96095064064,"internally_reserved":0,"allocated":264794112,"data_stored":264629123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746956,"kb_used_data":350204,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93624884,"statfs":{"total":96636764160,"available":95871881216,"internally_reserved":0,"allocated":358608896,"data_stored":358443907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357310464,"data_stored":357310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264511488,"data_stored":264511488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359596032,"data_stored":359596032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356831232,"data_stored":356831232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:17.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:17.969 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:17.969 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:18.001 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-09-15T21:08:16.022316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":715306798,"num_objects":87323,"num_object_clones":0,"num_object_copies":349292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107089,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113002,"ondisk_log_size":113002,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2770300,"kb_used_data":1310000,"kb_used_omap":14,"kb_used_meta":1460209,"kb_avail":374717060,"statfs":{"total":386547056640,"available":383710269440,"internally_reserved":0,"allocated":1341440000,"data_stored":1340780044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1495254578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,8,10,9,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28860416,"num_objects":-3523,"num_object_clones":0,"num_object_copies":-14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3523,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004611"},"pg_stats":[{"pgid":"1.f","version":"33'6985","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.166766+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:15.166766+0000","last_peered":"2024-09-15T21:08:15.166766+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:15.166766+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6985,"log_dups_size":0,"ondisk_log_size":6985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43966464,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7084","reported_seq":8661,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.794917+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:14.794917+0000","last_peered":"2024-09-15T21:08:14.794917+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:14.794917+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44791052,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'6989","reported_seq":7069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.434127+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:12.434127+0000","last_peered":"2024-09-15T21:08:12.434127+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:12.434127+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6989,"log_dups_size":0,"ondisk_log_size":6989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44587678,"num_objects":5444,"num_object_clones":0,"num_object_copies":21776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6988,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3186}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7039","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.752949+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:14.752949+0000","last_peered":"2024-09-15T21:08:14.752949+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:14.752949+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373260,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7150","reported_seq":7206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.891002+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:14.891002+0000","last_peered":"2024-09-15T21:08:14.891002+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:14.891002+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45126924,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6996","reported_seq":7041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122325+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:15.122325+0000","last_peered":"2024-09-15T21:08:15.122325+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:15.122325+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44072960,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7230","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.677659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:14.677659+0000","last_peered":"2024-09-15T21:08:14.677659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:14.677659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397260,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7211","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.530990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:14.530990+0000","last_peered":"2024-09-15T21:08:14.530990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:14.530990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45462796,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7058","reported_seq":7118,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122718+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:15.122718+0000","last_peered":"2024-09-15T21:08:15.122718+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:15.122718+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44182040,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7051","reported_seq":7132,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.424751+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:12.424751+0000","last_peered":"2024-09-15T21:08:12.424751+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:12.424751+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7051,"log_dups_size":0,"ondisk_log_size":7051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45143308,"num_objects":5511,"num_object_clones":0,"num_object_copies":22044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3176}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6997","reported_seq":11660,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:15.075659+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:15.075659+0000","last_peered":"2024-09-15T21:08:15.075659+0000","last_clean":"2024-09-15T21:08:15.075659+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:15.075659+0000","last_undegraded":"2024-09-15T21:08:15.075659+0000","last_fullsized":"2024-09-15T21:08:15.075659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1108,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6988","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.164659+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:15.164659+0000","last_peered":"2024-09-15T21:08:15.164659+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:15.164659+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44554886,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6987,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7014","reported_seq":7094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.297091+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:12.297091+0000","last_peered":"2024-09-15T21:08:12.297091+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:12.297091+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7014,"log_dups_size":0,"ondisk_log_size":7014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44948102,"num_objects":5487,"num_object_clones":0,"num_object_copies":21948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3201}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6999","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.115113+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:15.115113+0000","last_peered":"2024-09-15T21:08:15.115113+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:15.115113+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44504332,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7101","reported_seq":7184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:12.281133+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:12.281133+0000","last_peered":"2024-09-15T21:08:12.281133+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:12.281133+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7101,"log_dups_size":0,"ondisk_log_size":7101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45708556,"num_objects":5580,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7099,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3299}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7110","reported_seq":7179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.741608+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:14.741608+0000","last_peered":"2024-09-15T21:08:14.741608+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:14.741608+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553484,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":715306798,"num_objects":87323,"num_object_clones":0,"num_object_copies":349292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87323,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107089,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1338249216,"data_stored":1338249216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113002,"ondisk_log_size":113002,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754556,"kb_used_data":349416,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93617284,"statfs":{"total":96636764160,"available":95864098816,"internally_reserved":0,"allocated":357801984,"data_stored":357636995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739784,"kb_used_data":351792,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93632056,"statfs":{"total":96636764160,"available":95879225344,"internally_reserved":0,"allocated":360235008,"data_stored":360070019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529004,"kb_used_data":258588,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93842836,"statfs":{"total":96636764160,"available":96095064064,"internally_reserved":0,"allocated":264794112,"data_stored":264629123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746956,"kb_used_data":350204,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93624884,"statfs":{"total":96636764160,"available":95871881216,"internally_reserved":0,"allocated":358608896,"data_stored":358443907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357310464,"data_stored":357310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264511488,"data_stored":264511488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":359596032,"data_stored":359596032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356831232,"data_stored":356831232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:18.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:18.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'6985", 'reported_seq': 7031, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:15.166766+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:15.166766+0000', 'last_peered': '2024-09-15T21:08:15.166766+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:15.166766+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6985, 'log_dups_size': 0, 'ondisk_log_size': 6985, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43966464, 'num_objects': 5367, 'num_object_clones': 0, 'num_object_copies': 21468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6985, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3091}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.002 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:18.002 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7084", 'reported_seq': 8661, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.794917+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:14.794917+0000', 'last_peered': '2024-09-15T21:08:14.794917+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:14.794917+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7084, 'log_dups_size': 0, 'ondisk_log_size': 7084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44791052, 'num_objects': 5468, 'num_object_clones': 0, 'num_object_copies': 21872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1626, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7082, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 756, 'num_bytes_recovered': 6176768, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3842}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'6989", 'reported_seq': 7069, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:12.434127+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:12.434127+0000', 'last_peered': '2024-09-15T21:08:12.434127+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:12.434127+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6989, 'log_dups_size': 0, 'ondisk_log_size': 6989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44587678, 'num_objects': 5444, 'num_object_clones': 0, 'num_object_copies': 21776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5444, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 6988, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3186}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7039", 'reported_seq': 7108, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.752949+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:14.752949+0000', 'last_peered': '2024-09-15T21:08:14.752949+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:14.752949+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7039, 'log_dups_size': 0, 'ondisk_log_size': 7039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44373260, 'num_objects': 5417, 'num_object_clones': 0, 'num_object_copies': 21668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5417, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7037, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7150", 'reported_seq': 7206, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.891002+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:14.891002+0000', 'last_peered': '2024-09-15T21:08:14.891002+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:14.891002+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7150, 'log_dups_size': 0, 'ondisk_log_size': 7150, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45126924, 'num_objects': 5509, 'num_object_clones': 0, 'num_object_copies': 22036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7148, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3198}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'6996", 'reported_seq': 7041, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:15.122325+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:15.122325+0000', 'last_peered': '2024-09-15T21:08:15.122325+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:15.122325+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6996, 'log_dups_size': 0, 'ondisk_log_size': 6996, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44072960, 'num_objects': 5380, 'num_object_clones': 0, 'num_object_copies': 21520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6996, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3064}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:18.003 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7230", 'reported_seq': 7299, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.677659+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:14.677659+0000', 'last_peered': '2024-09-15T21:08:14.677659+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:14.677659+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7230, 'log_dups_size': 0, 'ondisk_log_size': 7230, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45397260, 'num_objects': 5542, 'num_object_clones': 0, 'num_object_copies': 22168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5542, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7228, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3176}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7211", 'reported_seq': 7280, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.530990+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:14.530990+0000', 'last_peered': '2024-09-15T21:08:14.530990+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:14.530990+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7211, 'log_dups_size': 0, 'ondisk_log_size': 7211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45462796, 'num_objects': 5550, 'num_object_clones': 0, 'num_object_copies': 22200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5550, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7209, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7058", 'reported_seq': 7118, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:15.122718+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:15.122718+0000', 'last_peered': '2024-09-15T21:08:15.122718+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:15.122718+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7058, 'log_dups_size': 0, 'ondisk_log_size': 7058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44182040, 'num_objects': 5394, 'num_object_clones': 0, 'num_object_copies': 21576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5394, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7055, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7051", 'reported_seq': 7132, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:12.424751+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:12.424751+0000', 'last_peered': '2024-09-15T21:08:12.424751+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:12.424751+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7051, 'log_dups_size': 0, 'ondisk_log_size': 7051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45143308, 'num_objects': 5511, 'num_object_clones': 0, 'num_object_copies': 22044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5511, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7049, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3176}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'6988", 'reported_seq': 7028, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:15.164659+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:15.164659+0000', 'last_peered': '2024-09-15T21:08:15.164659+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:15.164659+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6988, 'log_dups_size': 0, 'ondisk_log_size': 6988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44554886, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6987, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3183}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7014", 'reported_seq': 7094, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:12.297091+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:12.297091+0000', 'last_peered': '2024-09-15T21:08:12.297091+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:12.297091+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7014, 'log_dups_size': 0, 'ondisk_log_size': 7014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44948102, 'num_objects': 5487, 'num_object_clones': 0, 'num_object_copies': 21948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7013, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:18.004 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'6999", 'reported_seq': 7056, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:15.115113+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:15.115113+0000', 'last_peered': '2024-09-15T21:08:15.115113+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:15.115113+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 6999, 'log_dups_size': 0, 'ondisk_log_size': 6999, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44504332, 'num_objects': 5433, 'num_object_clones': 0, 'num_object_copies': 21732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 6997, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3154}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7101", 'reported_seq': 7184, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:12.281133+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:12.281133+0000', 'last_peered': '2024-09-15T21:08:12.281133+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:12.281133+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7101, 'log_dups_size': 0, 'ondisk_log_size': 7101, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45708556, 'num_objects': 5580, 'num_object_clones': 0, 'num_object_copies': 22320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7099, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:18.005 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7110", 'reported_seq': 7179, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:14.741608+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:14.741608+0000', 'last_peered': '2024-09-15T21:08:14.741608+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:14.741608+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7110, 'log_dups_size': 0, 'ondisk_log_size': 7110, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44553484, 'num_objects': 5439, 'num_object_clones': 0, 'num_object_copies': 21756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5439, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7108, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:18.006 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:18.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:18.794 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34067/348024 objects degraded (9.789%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":364,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":87006,"data_bytes":712709934,"bytes_used":2835845120,"bytes_avail":383711211520,"bytes_total":386547056640,"degraded_objects":34067,"degraded_total":348024,"degraded_ratio":0.09788692733834449,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":62780,"recovering_keys_per_sec":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:18.816 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:18.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:19.432 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:19.432 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:19.451 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-15T21:08:18.023091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":712709934,"num_objects":87006,"num_object_clones":0,"num_object_copies":348024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107406,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113319,"ondisk_log_size":113319,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769380,"kb_used_data":1304920,"kb_used_omap":14,"kb_used_meta":1464369,"kb_avail":374717980,"statfs":{"total":386547056640,"available":383711211520,"internally_reserved":0,"allocated":1336238080,"data_stored":1335578124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1499514418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,12,7,15,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23494656,"num_objects":-2868,"num_object_clones":0,"num_object_copies":-11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2868,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004719"},"pg_stats":[{"pgid":"1.f","version":"33'6985","reported_seq":7031,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.166766+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:15.166766+0000","last_peered":"2024-09-15T21:08:15.166766+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:15.166766+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6985,"log_dups_size":0,"ondisk_log_size":6985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43966464,"num_objects":5367,"num_object_clones":0,"num_object_copies":21468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6985,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3091}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7084","reported_seq":8661,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.794917+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:14.794917+0000","last_peered":"2024-09-15T21:08:14.794917+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:14.794917+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7084,"log_dups_size":0,"ondisk_log_size":7084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44791052,"num_objects":5468,"num_object_clones":0,"num_object_copies":21872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3842}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7056","reported_seq":7136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.374015+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:17.374015+0000","last_peered":"2024-09-15T21:08:17.374015+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:17.374015+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44038814,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7039","reported_seq":7108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.752949+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:14.752949+0000","last_peered":"2024-09-15T21:08:14.752949+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:14.752949+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44373260,"num_objects":5417,"num_object_clones":0,"num_object_copies":21668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7037,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7150","reported_seq":7206,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.891002+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:14.891002+0000","last_peered":"2024-09-15T21:08:14.891002+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:14.891002+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45126924,"num_objects":5509,"num_object_clones":0,"num_object_copies":22036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7148,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3198}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'6996","reported_seq":7041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122325+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:15.122325+0000","last_peered":"2024-09-15T21:08:15.122325+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:15.122325+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44072960,"num_objects":5380,"num_object_clones":0,"num_object_copies":21520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6996,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3064}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7230","reported_seq":7299,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.677659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:14.677659+0000","last_peered":"2024-09-15T21:08:14.677659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:14.677659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45397260,"num_objects":5542,"num_object_clones":0,"num_object_copies":22168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3176}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7211","reported_seq":7280,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.530990+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:14.530990+0000","last_peered":"2024-09-15T21:08:14.530990+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:14.530990+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7211,"log_dups_size":0,"ondisk_log_size":7211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45462796,"num_objects":5550,"num_object_clones":0,"num_object_copies":22200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7058","reported_seq":7118,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.122718+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:15.122718+0000","last_peered":"2024-09-15T21:08:15.122718+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:15.122718+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44182040,"num_objects":5394,"num_object_clones":0,"num_object_copies":21576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7055,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7146","reported_seq":7227,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.498224+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:17.498224+0000","last_peered":"2024-09-15T21:08:17.498224+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:17.498224+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44365068,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'6997","reported_seq":11660,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:15.075659+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:15.075659+0000","last_peered":"2024-09-15T21:08:15.075659+0000","last_clean":"2024-09-15T21:08:15.075659+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:15.075659+0000","last_undegraded":"2024-09-15T21:08:15.075659+0000","last_fullsized":"2024-09-15T21:08:15.075659+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":6997,"log_dups_size":0,"ondisk_log_size":6997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43933696,"num_objects":5363,"num_object_clones":0,"num_object_copies":21452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1108,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'6988","reported_seq":7028,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.164659+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:15.164659+0000","last_peered":"2024-09-15T21:08:15.164659+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:15.164659+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6988,"log_dups_size":0,"ondisk_log_size":6988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44554886,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6987,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3183}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7096","reported_seq":7176,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.426142+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:17.426142+0000","last_peered":"2024-09-15T21:08:17.426142+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:17.426142+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44276358,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'6999","reported_seq":7056,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:15.115113+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:15.115113+0000","last_peered":"2024-09-15T21:08:15.115113+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:15.115113+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44504332,"num_objects":5433,"num_object_clones":0,"num_object_copies":21732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6997,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3154}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7174","reported_seq":7257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.347168+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:17.347168+0000","last_peered":"2024-09-15T21:08:17.347168+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:17.347168+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110540,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7110","reported_seq":7179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:14.741608+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:14.741608+0000","last_peered":"2024-09-15T21:08:14.741608+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:14.741608+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44553484,"num_objects":5439,"num_object_clones":0,"num_object_copies":21756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":712709934,"num_objects":87006,"num_object_clones":0,"num_object_copies":348024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":107406,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":778,"num_bytes_recovered":6176768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1333030912,"data_stored":1333030912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113319,"ondisk_log_size":113319,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":754556,"kb_used_data":349416,"kb_used_omap":4,"kb_used_meta":405115,"kb_avail":93617284,"statfs":{"total":96636764160,"available":95864098816,"internally_reserved":0,"allocated":357801984,"data_stored":357636995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":414838480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,3,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738864,"kb_used_data":346712,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93632976,"statfs":{"total":96636764160,"available":95880167424,"internally_reserved":0,"allocated":355033088,"data_stored":354868099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051636,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529004,"kb_used_data":258588,"kb_used_omap":4,"kb_used_meta":270395,"kb_avail":93842836,"statfs":{"total":96636764160,"available":96095064064,"internally_reserved":0,"allocated":264794112,"data_stored":264629123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":276885199},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746956,"kb_used_data":350204,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93624884,"statfs":{"total":96636764160,"available":95871881216,"internally_reserved":0,"allocated":358608896,"data_stored":358443907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,0,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357310464,"data_stored":357310464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264511488,"data_stored":264511488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354377728,"data_stored":354377728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356831232,"data_stored":356831232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:22.453 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:23.032 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:23.033 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:23.054 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-15T21:08:22.024701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705279790,"num_objects":86099,"num_object_clones":0,"num_object_copies":344396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108313,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114226,"ondisk_log_size":114226,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772856,"kb_used_data":1294476,"kb_used_omap":14,"kb_used_meta":1478257,"kb_avail":374714504,"statfs":{"total":386547056640,"available":383707652096,"internally_reserved":0,"allocated":1325543424,"data_stored":1324883468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1513735730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,10,6,14,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25108480,"num_objects":-3065,"num_object_clones":0,"num_object_copies":-12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004669"},"pg_stats":[{"pgid":"1.f","version":"33'7046","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.298086+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:20.298086+0000","last_peered":"2024-09-15T21:08:20.298086+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:20.298086+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7046,"log_dups_size":0,"ondisk_log_size":7046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7162","reported_seq":8829,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.717363+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:19.717363+0000","last_peered":"2024-09-15T21:08:19.717363+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:19.717363+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7162,"log_dups_size":0,"ondisk_log_size":7162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152076,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7160,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7056","reported_seq":7136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.374015+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:17.374015+0000","last_peered":"2024-09-15T21:08:17.374015+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:17.374015+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44038814,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7103","reported_seq":7172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.670416+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:19.670416+0000","last_peered":"2024-09-15T21:08:19.670416+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:19.670416+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7103,"log_dups_size":0,"ondisk_log_size":7103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7235","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.180335+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:20.180335+0000","last_peered":"2024-09-15T21:08:20.180335+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:20.180335+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44430604,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7082","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.259699+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:20.259699+0000","last_peered":"2024-09-15T21:08:20.259699+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:20.259699+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43368448,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7306","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.479741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:19.479741+0000","last_peered":"2024-09-15T21:08:19.479741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:19.479741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7288","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.731448+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:19.731448+0000","last_peered":"2024-09-15T21:08:19.731448+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:19.731448+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44832012,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7149","reported_seq":7209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.247375+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:20.247375+0000","last_peered":"2024-09-15T21:08:20.247375+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:20.247375+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43436568,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7146","reported_seq":7227,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.498224+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:17.498224+0000","last_peered":"2024-09-15T21:08:17.498224+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:17.498224+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44365068,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7077","reported_seq":11739,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:20.182713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:20.182713+0000","last_peered":"2024-09-15T21:08:20.182713+0000","last_clean":"2024-09-15T21:08:20.182713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:20.182713+0000","last_undegraded":"2024-09-15T21:08:20.182713+0000","last_fullsized":"2024-09-15T21:08:20.182713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43278336,"num_objects":5283,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1188,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7061","reported_seq":7101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.231178+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:20.231178+0000","last_peered":"2024-09-15T21:08:20.231178+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:20.231178+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43956870,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3110}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7096","reported_seq":7176,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.426142+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:17.426142+0000","last_peered":"2024-09-15T21:08:17.426142+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:17.426142+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44276358,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7067","reported_seq":7124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.282345+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:20.282345+0000","last_peered":"2024-09-15T21:08:20.282345+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:20.282345+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43947276,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7174","reported_seq":7257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.347168+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:17.347168+0000","last_peered":"2024-09-15T21:08:17.347168+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:17.347168+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110540,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7178","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.483973+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:19.483973+0000","last_peered":"2024-09-15T21:08:19.483973+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:19.483973+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7178,"log_dups_size":0,"ondisk_log_size":7178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996428,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705279790,"num_objects":86099,"num_object_clones":0,"num_object_copies":344396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108313,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1322409984,"data_stored":1322409984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114226,"ondisk_log_size":114226,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737052,"kb_used_data":344388,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93634788,"statfs":{"total":96636764160,"available":95882022912,"internally_reserved":0,"allocated":352653312,"data_stored":352488323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738864,"kb_used_data":346712,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93632976,"statfs":{"total":96636764160,"available":95880167424,"internally_reserved":0,"allocated":355033088,"data_stored":354868099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550916,"kb_used_data":258292,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93820924,"statfs":{"total":96636764160,"available":96072626176,"internally_reserved":0,"allocated":264491008,"data_stored":264326019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746024,"kb_used_data":345084,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93625816,"statfs":{"total":96636764160,"available":95872835584,"internally_reserved":0,"allocated":353366016,"data_stored":353201027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352182272,"data_stored":352182272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264216576,"data_stored":264216576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354377728,"data_stored":354377728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351633408,"data_stored":351633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:23.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:23.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:23.660 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:23.688 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-15T21:08:22.024701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":705279790,"num_objects":86099,"num_object_clones":0,"num_object_copies":344396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108313,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114226,"ondisk_log_size":114226,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772856,"kb_used_data":1294476,"kb_used_omap":14,"kb_used_meta":1478257,"kb_avail":374714504,"statfs":{"total":386547056640,"available":383707652096,"internally_reserved":0,"allocated":1325543424,"data_stored":1324883468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1513735730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,10,6,14,10],"upper_bound":128},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25108480,"num_objects":-3065,"num_object_clones":0,"num_object_copies":-12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3065,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004669"},"pg_stats":[{"pgid":"1.f","version":"33'7046","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.298086+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:20.298086+0000","last_peered":"2024-09-15T21:08:20.298086+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:20.298086+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7046,"log_dups_size":0,"ondisk_log_size":7046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7162","reported_seq":8829,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.717363+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:19.717363+0000","last_peered":"2024-09-15T21:08:19.717363+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:19.717363+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7162,"log_dups_size":0,"ondisk_log_size":7162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152076,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7160,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7056","reported_seq":7136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.374015+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:17.374015+0000","last_peered":"2024-09-15T21:08:17.374015+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:17.374015+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44038814,"num_objects":5377,"num_object_clones":0,"num_object_copies":21508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7055,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3119}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7103","reported_seq":7172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.670416+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:19.670416+0000","last_peered":"2024-09-15T21:08:19.670416+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:19.670416+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7103,"log_dups_size":0,"ondisk_log_size":7103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7235","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.180335+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:20.180335+0000","last_peered":"2024-09-15T21:08:20.180335+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:20.180335+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44430604,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7082","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.259699+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:20.259699+0000","last_peered":"2024-09-15T21:08:20.259699+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:20.259699+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43368448,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7306","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.479741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:19.479741+0000","last_peered":"2024-09-15T21:08:19.479741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:19.479741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7288","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.731448+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:19.731448+0000","last_peered":"2024-09-15T21:08:19.731448+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:19.731448+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44832012,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7149","reported_seq":7209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.247375+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:20.247375+0000","last_peered":"2024-09-15T21:08:20.247375+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:20.247375+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43436568,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7146","reported_seq":7227,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.498224+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:17.498224+0000","last_peered":"2024-09-15T21:08:17.498224+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:17.498224+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44365068,"num_objects":5416,"num_object_clones":0,"num_object_copies":21664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":3081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7077","reported_seq":11739,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:20.182713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:20.182713+0000","last_peered":"2024-09-15T21:08:20.182713+0000","last_clean":"2024-09-15T21:08:20.182713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:20.182713+0000","last_undegraded":"2024-09-15T21:08:20.182713+0000","last_fullsized":"2024-09-15T21:08:20.182713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43278336,"num_objects":5283,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1188,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7061","reported_seq":7101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.231178+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:20.231178+0000","last_peered":"2024-09-15T21:08:20.231178+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:20.231178+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43956870,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3110}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7096","reported_seq":7176,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.426142+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:17.426142+0000","last_peered":"2024-09-15T21:08:17.426142+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:17.426142+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7096,"log_dups_size":0,"ondisk_log_size":7096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44276358,"num_objects":5405,"num_object_clones":0,"num_object_copies":21620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7067","reported_seq":7124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.282345+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:20.282345+0000","last_peered":"2024-09-15T21:08:20.282345+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:20.282345+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43947276,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7174","reported_seq":7257,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:17.347168+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:17.347168+0000","last_peered":"2024-09-15T21:08:17.347168+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:17.347168+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7174,"log_dups_size":0,"ondisk_log_size":7174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45110540,"num_objects":5507,"num_object_clones":0,"num_object_copies":22028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7172,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7178","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.483973+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:19.483973+0000","last_peered":"2024-09-15T21:08:19.483973+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:19.483973+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7178,"log_dups_size":0,"ondisk_log_size":7178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996428,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":705279790,"num_objects":86099,"num_object_clones":0,"num_object_copies":344396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86099,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108313,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1322409984,"data_stored":1322409984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114226,"ondisk_log_size":114226,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737052,"kb_used_data":344388,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93634788,"statfs":{"total":96636764160,"available":95882022912,"internally_reserved":0,"allocated":352653312,"data_stored":352488323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738864,"kb_used_data":346712,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93632976,"statfs":{"total":96636764160,"available":95880167424,"internally_reserved":0,"allocated":355033088,"data_stored":354868099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051637,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550916,"kb_used_data":258292,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93820924,"statfs":{"total":96636764160,"available":96072626176,"internally_reserved":0,"allocated":264491008,"data_stored":264326019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746024,"kb_used_data":345084,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93625816,"statfs":{"total":96636764160,"available":95872835584,"internally_reserved":0,"allocated":353366016,"data_stored":353201027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352182272,"data_stored":352182272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":264216576,"data_stored":264216576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":354377728,"data_stored":354377728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351633408,"data_stored":351633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7046", 'reported_seq': 7092, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.298086+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:20.298086+0000', 'last_peered': '2024-09-15T21:08:20.298086+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:20.298086+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7046, 'log_dups_size': 0, 'ondisk_log_size': 7046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43466752, 'num_objects': 5306, 'num_object_clones': 0, 'num_object_copies': 21224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7046, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 3030}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7162", 'reported_seq': 8829, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:19.717363+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:19.717363+0000', 'last_peered': '2024-09-15T21:08:19.717363+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:19.717363+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7162, 'log_dups_size': 0, 'ondisk_log_size': 7162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44152076, 'num_objects': 5390, 'num_object_clones': 0, 'num_object_copies': 21560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1581, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7160, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 801, 'num_bytes_recovered': 6545408, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3809}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7056", 'reported_seq': 7136, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:17.374015+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:17.374015+0000', 'last_peered': '2024-09-15T21:08:17.374015+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:17.374015+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7056, 'log_dups_size': 0, 'ondisk_log_size': 7056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44038814, 'num_objects': 5377, 'num_object_clones': 0, 'num_object_copies': 21508, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5377, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7055, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 3119}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7103", 'reported_seq': 7172, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:19.670416+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:19.670416+0000', 'last_peered': '2024-09-15T21:08:19.670416+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:19.670416+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7103, 'log_dups_size': 0, 'ondisk_log_size': 7103, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43848972, 'num_objects': 5353, 'num_object_clones': 0, 'num_object_copies': 21412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7101, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3019}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7235", 'reported_seq': 7291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.180335+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:20.180335+0000', 'last_peered': '2024-09-15T21:08:20.180335+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:20.180335+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7235, 'log_dups_size': 0, 'ondisk_log_size': 7235, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44430604, 'num_objects': 5424, 'num_object_clones': 0, 'num_object_copies': 21696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7233, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3113}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7082", 'reported_seq': 7127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.259699+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:20.259699+0000', 'last_peered': '2024-09-15T21:08:20.259699+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:20.259699+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7082, 'log_dups_size': 0, 'ondisk_log_size': 7082, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43368448, 'num_objects': 5294, 'num_object_clones': 0, 'num_object_copies': 21176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5294, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7082, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2978}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7306", 'reported_seq': 7376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:19.479741+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:19.479741+0000', 'last_peered': '2024-09-15T21:08:19.479741+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:19.479741+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44774668, 'num_objects': 5466, 'num_object_clones': 0, 'num_object_copies': 21864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7304, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3100}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7288", 'reported_seq': 7357, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:19.731448+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:19.731448+0000', 'last_peered': '2024-09-15T21:08:19.731448+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:19.731448+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7288, 'log_dups_size': 0, 'ondisk_log_size': 7288, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44832012, 'num_objects': 5473, 'num_object_clones': 0, 'num_object_copies': 21892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5473, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7286, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2999}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7149", 'reported_seq': 7209, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.247375+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:20.247375+0000', 'last_peered': '2024-09-15T21:08:20.247375+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:20.247375+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7149, 'log_dups_size': 0, 'ondisk_log_size': 7149, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43436568, 'num_objects': 5303, 'num_object_clones': 0, 'num_object_copies': 21212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5303, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7146, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2971}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7146", 'reported_seq': 7227, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:17.498224+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:17.498224+0000', 'last_peered': '2024-09-15T21:08:17.498224+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:17.498224+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7146, 'log_dups_size': 0, 'ondisk_log_size': 7146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44365068, 'num_objects': 5416, 'num_object_clones': 0, 'num_object_copies': 21664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5416, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7144, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 3081}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7061", 'reported_seq': 7101, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.231178+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:20.231178+0000', 'last_peered': '2024-09-15T21:08:20.231178+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:20.231178+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7061, 'log_dups_size': 0, 'ondisk_log_size': 7061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43956870, 'num_objects': 5366, 'num_object_clones': 0, 'num_object_copies': 21464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5366, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7060, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3110}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7096", 'reported_seq': 7176, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:17.426142+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:17.426142+0000', 'last_peered': '2024-09-15T21:08:17.426142+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:17.426142+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7096, 'log_dups_size': 0, 'ondisk_log_size': 7096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44276358, 'num_objects': 5405, 'num_object_clones': 0, 'num_object_copies': 21620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7095, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3119}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7067", 'reported_seq': 7124, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:20.282345+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:20.282345+0000', 'last_peered': '2024-09-15T21:08:20.282345+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:20.282345+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7067, 'log_dups_size': 0, 'ondisk_log_size': 7067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43947276, 'num_objects': 5365, 'num_object_clones': 0, 'num_object_copies': 21460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7065, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:23.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:23.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7174", 'reported_seq': 7257, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:17.347168+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:17.347168+0000', 'last_peered': '2024-09-15T21:08:17.347168+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:17.347168+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7174, 'log_dups_size': 0, 'ondisk_log_size': 7174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 45110540, 'num_objects': 5507, 'num_object_clones': 0, 'num_object_copies': 22028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7172, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:23.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:23.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7178", 'reported_seq': 7247, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:19.483973+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:19.483973+0000', 'last_peered': '2024-09-15T21:08:19.483973+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:19.483973+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7178, 'log_dups_size': 0, 'ondisk_log_size': 7178, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43996428, 'num_objects': 5371, 'num_object_clones': 0, 'num_object_copies': 21484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7176, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 3056}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:23.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:24.418 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:24.418 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 34022/343052 objects degraded (9.917%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":370,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":85763,"data_bytes":702527278,"bytes_used":2825392128,"bytes_avail":383721664512,"bytes_total":386547056640,"degraded_objects":34022,"degraded_total":343052,"degraded_ratio":0.099174469176684585,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":63463,"recovering_keys_per_sec":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":236},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:24.446 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:24.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:25.036 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:25.037 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:25.058 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-15T21:08:24.025332+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":702527278,"num_objects":85763,"num_object_clones":0,"num_object_copies":343052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108649,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114562,"ondisk_log_size":114562,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759172,"kb_used_data":1289120,"kb_used_omap":14,"kb_used_meta":1469937,"kb_avail":374728188,"statfs":{"total":386547056640,"available":383721664512,"internally_reserved":0,"allocated":1320058880,"data_stored":1319398924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1505216050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,8,2,18,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23306240,"num_objects":-2845,"num_object_clones":0,"num_object_copies":-11380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-93,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2845,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004626"},"pg_stats":[{"pgid":"1.f","version":"33'7046","reported_seq":7092,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.298086+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:20.298086+0000","last_peered":"2024-09-15T21:08:20.298086+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:20.298086+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7046,"log_dups_size":0,"ondisk_log_size":7046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43466752,"num_objects":5306,"num_object_clones":0,"num_object_copies":21224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7046,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":3030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7162","reported_seq":8829,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.717363+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:19.717363+0000","last_peered":"2024-09-15T21:08:19.717363+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:19.717363+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7162,"log_dups_size":0,"ondisk_log_size":7162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44152076,"num_objects":5390,"num_object_clones":0,"num_object_copies":21560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7160,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":801,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3809}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7133","reported_seq":7213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:22.360232+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:22.360232+0000","last_peered":"2024-09-15T21:08:22.360232+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:22.360232+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7133,"log_dups_size":0,"ondisk_log_size":7133,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43408030,"num_objects":5300,"num_object_clones":0,"num_object_copies":21200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7132,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":3042}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7103","reported_seq":7172,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.670416+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:19.670416+0000","last_peered":"2024-09-15T21:08:19.670416+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:19.670416+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7103,"log_dups_size":0,"ondisk_log_size":7103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7101,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3019}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7235","reported_seq":7291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.180335+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:20.180335+0000","last_peered":"2024-09-15T21:08:20.180335+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:20.180335+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44430604,"num_objects":5424,"num_object_clones":0,"num_object_copies":21696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7233,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7082","reported_seq":7127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.259699+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:20.259699+0000","last_peered":"2024-09-15T21:08:20.259699+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:20.259699+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7082,"log_dups_size":0,"ondisk_log_size":7082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43368448,"num_objects":5294,"num_object_clones":0,"num_object_copies":21176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2978}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7306","reported_seq":7376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.479741+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:19.479741+0000","last_peered":"2024-09-15T21:08:19.479741+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:19.479741+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44774668,"num_objects":5466,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3100}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7288","reported_seq":7357,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.731448+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:19.731448+0000","last_peered":"2024-09-15T21:08:19.731448+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:19.731448+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7288,"log_dups_size":0,"ondisk_log_size":7288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44832012,"num_objects":5473,"num_object_clones":0,"num_object_copies":21892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7286,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2999}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7149","reported_seq":7209,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.247375+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:20.247375+0000","last_peered":"2024-09-15T21:08:20.247375+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:20.247375+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43436568,"num_objects":5303,"num_object_clones":0,"num_object_copies":21212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2971}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7230","reported_seq":7311,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:22.550503+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:22.550503+0000","last_peered":"2024-09-15T21:08:22.550503+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:22.550503+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43676940,"num_objects":5332,"num_object_clones":0,"num_object_copies":21328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7228,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2997}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7077","reported_seq":11739,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:20.182713+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:20.182713+0000","last_peered":"2024-09-15T21:08:20.182713+0000","last_clean":"2024-09-15T21:08:20.182713+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:20.182713+0000","last_undegraded":"2024-09-15T21:08:20.182713+0000","last_fullsized":"2024-09-15T21:08:20.182713+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7077,"log_dups_size":0,"ondisk_log_size":7077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43278336,"num_objects":5283,"num_object_clones":0,"num_object_copies":21132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1188,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7061","reported_seq":7101,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.231178+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:20.231178+0000","last_peered":"2024-09-15T21:08:20.231178+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:20.231178+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43956870,"num_objects":5366,"num_object_clones":0,"num_object_copies":21464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3110}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7176","reported_seq":7255,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:22.488317+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:22.488317+0000","last_peered":"2024-09-15T21:08:22.488317+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:22.488317+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7176,"log_dups_size":0,"ondisk_log_size":7176,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43620998,"num_objects":5325,"num_object_clones":0,"num_object_copies":21300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7175,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3039}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7067","reported_seq":7124,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:20.282345+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:20.282345+0000","last_peered":"2024-09-15T21:08:20.282345+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:20.282345+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43947276,"num_objects":5365,"num_object_clones":0,"num_object_copies":21460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7065,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7269","reported_seq":7350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:22.481281+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:22.481281+0000","last_peered":"2024-09-15T21:08:22.481281+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:22.481281+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7269,"log_dups_size":0,"ondisk_log_size":7269,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44332300,"num_objects":5412,"num_object_clones":0,"num_object_copies":21648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7267,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3131}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7178","reported_seq":7247,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:19.483973+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:19.483973+0000","last_peered":"2024-09-15T21:08:19.483973+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:19.483973+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7178,"log_dups_size":0,"ondisk_log_size":7178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43996428,"num_objects":5371,"num_object_clones":0,"num_object_copies":21484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":3056}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":702527278,"num_objects":85763,"num_object_clones":0,"num_object_copies":343052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34022,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108649,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":6545408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1316892672,"data_stored":1316892672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114562,"ondisk_log_size":114562,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737052,"kb_used_data":344388,"kb_used_omap":4,"kb_used_meta":392635,"kb_avail":93634788,"statfs":{"total":96636764160,"available":95882022912,"internally_reserved":0,"allocated":352653312,"data_stored":352488323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":402058960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,5,0,6,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737960,"kb_used_data":341656,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93633880,"statfs":{"total":96636764160,"available":95881093120,"internally_reserved":0,"allocated":349855744,"data_stored":349690755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538136,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93833704,"statfs":{"total":96636764160,"available":96085712896,"internally_reserved":0,"allocated":264183808,"data_stored":264018819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286846671},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":746024,"kb_used_data":345084,"kb_used_omap":4,"kb_used_meta":400891,"kb_avail":93625816,"statfs":{"total":96636764160,"available":95872835584,"internally_reserved":0,"allocated":353366016,"data_stored":353201027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":410513102},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":352182272,"data_stored":352182272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263905280,"data_stored":263905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":349171712,"data_stored":349171712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":351633408,"data_stored":351633408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:28.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:28.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:28.660 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:28.686 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-15T21:08:28.030485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692385582,"num_objects":84525,"num_object_clones":0,"num_object_copies":338100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109887,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115800,"ondisk_log_size":115800,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739992,"kb_used_data":1274128,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374747368,"statfs":{"total":386547056640,"available":383741304832,"internally_reserved":0,"allocated":1304707072,"data_stored":1304047116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1500956210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,12,17,15,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22921216,"num_objects":-2798,"num_object_clones":0,"num_object_copies":-11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008169"},"pg_stats":[{"pgid":"1.f","version":"33'7122","reported_seq":7168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.433138+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:25.433138+0000","last_peered":"2024-09-15T21:08:25.433138+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:25.433138+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42844160,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7250","reported_seq":9004,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.835835+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:24.835835+0000","last_peered":"2024-09-15T21:08:24.835835+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:24.835835+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431180,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":845,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7215","reported_seq":7294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.618090+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:27.618090+0000","last_peered":"2024-09-15T21:08:27.618090+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:27.618090+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7215,"log_dups_size":0,"ondisk_log_size":7215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736286,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7214,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2960}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7166","reported_seq":7235,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.851446+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:24.851446+0000","last_peered":"2024-09-15T21:08:24.851446+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:24.851446+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43332876,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7306","reported_seq":7362,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.343902+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:25.343902+0000","last_peered":"2024-09-15T21:08:25.343902+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:25.343902+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7155","reported_seq":7200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.350762+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:25.350762+0000","last_peered":"2024-09-15T21:08:25.350762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:25.350762+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42770432,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7393","reported_seq":7463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.653410+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:24.653410+0000","last_peered":"2024-09-15T21:08:24.653410+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:24.653410+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44061964,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3013}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7382","reported_seq":7451,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.721528+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:24.721528+0000","last_peered":"2024-09-15T21:08:24.721528+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:24.721528+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7382,"log_dups_size":0,"ondisk_log_size":7382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44061964,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7380,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7237","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.210717+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:25.210717+0000","last_peered":"2024-09-15T21:08:25.210717+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:25.210717+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42715672,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7284","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.446996+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:27.446996+0000","last_peered":"2024-09-15T21:08:27.446996+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:27.446996+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7282,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7150","reported_seq":11813,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:25.292597+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:25.292597+0000","last_peered":"2024-09-15T21:08:25.292597+0000","last_clean":"2024-09-15T21:08:25.292597+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:25.292597+0000","last_undegraded":"2024-09-15T21:08:25.292597+0000","last_fullsized":"2024-09-15T21:08:25.292597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42680320,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7143","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.321371+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:25.321371+0000","last_peered":"2024-09-15T21:08:25.321371+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:25.321371+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285126,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3028}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7249","reported_seq":7329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.484111+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:27.484111+0000","last_peered":"2024-09-15T21:08:27.484111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:27.484111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43022982,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7146","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.366787+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:25.366787+0000","last_peered":"2024-09-15T21:08:25.366787+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:25.366787+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43300108,"num_objects":5286,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7348","reported_seq":7431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.651077+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:27.651077+0000","last_peered":"2024-09-15T21:08:27.651077+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:27.651077+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7348,"log_dups_size":0,"ondisk_log_size":7348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43685132,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3052}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7254","reported_seq":7323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.807168+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:24.807168+0000","last_peered":"2024-09-15T21:08:24.807168+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:24.807168+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7254,"log_dups_size":0,"ondisk_log_size":7254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43373836,"num_objects":5295,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7252,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692385582,"num_objects":84525,"num_object_clones":0,"num_object_copies":338100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109887,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1301643264,"data_stored":1301643264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115800,"ondisk_log_size":115800,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736212,"kb_used_data":339392,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93635628,"statfs":{"total":96636764160,"available":95882883072,"internally_reserved":0,"allocated":347537408,"data_stored":347372419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,10,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737192,"kb_used_data":336728,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93634648,"statfs":{"total":96636764160,"available":95881879552,"internally_reserved":0,"allocated":344809472,"data_stored":344644483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538136,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93833704,"statfs":{"total":96636764160,"available":96085712896,"internally_reserved":0,"allocated":264183808,"data_stored":264018819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286846671},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728452,"kb_used_data":340016,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93643388,"statfs":{"total":96636764160,"available":95890829312,"internally_reserved":0,"allocated":348176384,"data_stored":348011395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":397733582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347115520,"data_stored":347115520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263905280,"data_stored":263905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344096768,"data_stored":344096768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346525696,"data_stored":346525696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:28.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:29.284 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:29.285 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:29.308 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2024-09-15T21:08:28.030485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":692385582,"num_objects":84525,"num_object_clones":0,"num_object_copies":338100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109887,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115800,"ondisk_log_size":115800,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739992,"kb_used_data":1274128,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374747368,"statfs":{"total":386547056640,"available":383741304832,"internally_reserved":0,"allocated":1304707072,"data_stored":1304047116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1500956210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,5,12,17,15,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22921216,"num_objects":-2798,"num_object_clones":0,"num_object_copies":-11192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2798,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008169"},"pg_stats":[{"pgid":"1.f","version":"33'7122","reported_seq":7168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.433138+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:25.433138+0000","last_peered":"2024-09-15T21:08:25.433138+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:25.433138+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42844160,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7250","reported_seq":9004,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.835835+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:24.835835+0000","last_peered":"2024-09-15T21:08:24.835835+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:24.835835+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43431180,"num_objects":5302,"num_object_clones":0,"num_object_copies":21208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1537,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":845,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7215","reported_seq":7294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.618090+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:27.618090+0000","last_peered":"2024-09-15T21:08:27.618090+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:27.618090+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7215,"log_dups_size":0,"ondisk_log_size":7215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736286,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7214,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2960}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7166","reported_seq":7235,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.851446+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:24.851446+0000","last_peered":"2024-09-15T21:08:24.851446+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:24.851446+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7166,"log_dups_size":0,"ondisk_log_size":7166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43332876,"num_objects":5290,"num_object_clones":0,"num_object_copies":21160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7164,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2956}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7306","reported_seq":7362,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.343902+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:25.343902+0000","last_peered":"2024-09-15T21:08:25.343902+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:25.343902+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7155","reported_seq":7200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.350762+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:25.350762+0000","last_peered":"2024-09-15T21:08:25.350762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:25.350762+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42770432,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7393","reported_seq":7463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.653410+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:24.653410+0000","last_peered":"2024-09-15T21:08:24.653410+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:24.653410+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7393,"log_dups_size":0,"ondisk_log_size":7393,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44061964,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":3013}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7382","reported_seq":7451,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.721528+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:24.721528+0000","last_peered":"2024-09-15T21:08:24.721528+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:24.721528+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7382,"log_dups_size":0,"ondisk_log_size":7382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44061964,"num_objects":5379,"num_object_clones":0,"num_object_copies":21516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7380,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7237","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.210717+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:25.210717+0000","last_peered":"2024-09-15T21:08:25.210717+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:25.210717+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42715672,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7284","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.446996+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:27.446996+0000","last_peered":"2024-09-15T21:08:27.446996+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:27.446996+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7282,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7150","reported_seq":11813,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:25.292597+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:25.292597+0000","last_peered":"2024-09-15T21:08:25.292597+0000","last_clean":"2024-09-15T21:08:25.292597+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:25.292597+0000","last_undegraded":"2024-09-15T21:08:25.292597+0000","last_fullsized":"2024-09-15T21:08:25.292597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42680320,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7143","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.321371+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:25.321371+0000","last_peered":"2024-09-15T21:08:25.321371+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:25.321371+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285126,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3028}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7249","reported_seq":7329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.484111+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:27.484111+0000","last_peered":"2024-09-15T21:08:27.484111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:27.484111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43022982,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7146","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.366787+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:25.366787+0000","last_peered":"2024-09-15T21:08:25.366787+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:25.366787+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43300108,"num_objects":5286,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7348","reported_seq":7431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.651077+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:27.651077+0000","last_peered":"2024-09-15T21:08:27.651077+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:27.651077+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7348,"log_dups_size":0,"ondisk_log_size":7348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43685132,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3052}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7254","reported_seq":7323,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:24.807168+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:24.807168+0000","last_peered":"2024-09-15T21:08:24.807168+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:24.807168+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7254,"log_dups_size":0,"ondisk_log_size":7254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43373836,"num_objects":5295,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7252,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":692385582,"num_objects":84525,"num_object_clones":0,"num_object_copies":338100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84525,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109887,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1301643264,"data_stored":1301643264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115800,"ondisk_log_size":115800,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736212,"kb_used_data":339392,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93635628,"statfs":{"total":96636764160,"available":95882883072,"internally_reserved":0,"allocated":347537408,"data_stored":347372419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,10,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737192,"kb_used_data":336728,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93634648,"statfs":{"total":96636764160,"available":95881879552,"internally_reserved":0,"allocated":344809472,"data_stored":344644483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051638,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538136,"kb_used_data":257992,"kb_used_omap":4,"kb_used_meta":280123,"kb_avail":93833704,"statfs":{"total":96636764160,"available":96085712896,"internally_reserved":0,"allocated":264183808,"data_stored":264018819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":286846671},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728452,"kb_used_data":340016,"kb_used_omap":4,"kb_used_meta":388411,"kb_avail":93643388,"statfs":{"total":96636764160,"available":95890829312,"internally_reserved":0,"allocated":348176384,"data_stored":348011395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":397733582},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":347115520,"data_stored":347115520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263905280,"data_stored":263905280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344096768,"data_stored":344096768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346525696,"data_stored":346525696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:29.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:29.309 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7122", 'reported_seq': 7168, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.433138+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:25.433138+0000', 'last_peered': '2024-09-15T21:08:25.433138+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:25.433138+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7122, 'log_dups_size': 0, 'ondisk_log_size': 7122, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42844160, 'num_objects': 5230, 'num_object_clones': 0, 'num_object_copies': 20920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5230, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7122, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2954}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.309 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7250", 'reported_seq': 9004, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:24.835835+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:24.835835+0000', 'last_peered': '2024-09-15T21:08:24.835835+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:24.835835+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7250, 'log_dups_size': 0, 'ondisk_log_size': 7250, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43431180, 'num_objects': 5302, 'num_object_clones': 0, 'num_object_copies': 21208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1537, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5302, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7248, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 845, 'num_bytes_recovered': 6905856, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7215", 'reported_seq': 7294, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:27.618090+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:27.618090+0000', 'last_peered': '2024-09-15T21:08:27.618090+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:27.618090+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7215, 'log_dups_size': 0, 'ondisk_log_size': 7215, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42736286, 'num_objects': 5218, 'num_object_clones': 0, 'num_object_copies': 20872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5218, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7214, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2960}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7166", 'reported_seq': 7235, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:24.851446+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:24.851446+0000', 'last_peered': '2024-09-15T21:08:24.851446+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:24.851446+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7166, 'log_dups_size': 0, 'ondisk_log_size': 7166, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43332876, 'num_objects': 5290, 'num_object_clones': 0, 'num_object_copies': 21160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7164, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2956}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7306", 'reported_seq': 7362, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.343902+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:25.343902+0000', 'last_peered': '2024-09-15T21:08:25.343902+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:25.343902+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7306, 'log_dups_size': 0, 'ondisk_log_size': 7306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43848972, 'num_objects': 5353, 'num_object_clones': 0, 'num_object_copies': 21412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7304, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 3042}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7155", 'reported_seq': 7200, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.350762+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:25.350762+0000', 'last_peered': '2024-09-15T21:08:25.350762+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:25.350762+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7155, 'log_dups_size': 0, 'ondisk_log_size': 7155, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42770432, 'num_objects': 5221, 'num_object_clones': 0, 'num_object_copies': 20884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5221, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7155, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2905}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:29.310 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7393", 'reported_seq': 7463, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:24.653410+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:24.653410+0000', 'last_peered': '2024-09-15T21:08:24.653410+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:24.653410+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7393, 'log_dups_size': 0, 'ondisk_log_size': 7393, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44061964, 'num_objects': 5379, 'num_object_clones': 0, 'num_object_copies': 21516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7391, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 3013}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7382", 'reported_seq': 7451, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:24.721528+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:24.721528+0000', 'last_peered': '2024-09-15T21:08:24.721528+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:24.721528+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7382, 'log_dups_size': 0, 'ondisk_log_size': 7382, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 44061964, 'num_objects': 5379, 'num_object_clones': 0, 'num_object_copies': 21516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7380, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7237", 'reported_seq': 7297, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.210717+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:25.210717+0000', 'last_peered': '2024-09-15T21:08:25.210717+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:25.210717+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7237, 'log_dups_size': 0, 'ondisk_log_size': 7237, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42715672, 'num_objects': 5215, 'num_object_clones': 0, 'num_object_copies': 20860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7234, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2883}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7284", 'reported_seq': 7365, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:27.446996+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:27.446996+0000', 'last_peered': '2024-09-15T21:08:27.446996+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:27.446996+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7284, 'log_dups_size': 0, 'ondisk_log_size': 7284, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43234572, 'num_objects': 5278, 'num_object_clones': 0, 'num_object_copies': 21112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7282, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2943}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7143", 'reported_seq': 7182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.321371+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:25.321371+0000', 'last_peered': '2024-09-15T21:08:25.321371+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:25.321371+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7143, 'log_dups_size': 0, 'ondisk_log_size': 7143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43285126, 'num_objects': 5284, 'num_object_clones': 0, 'num_object_copies': 21136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7142, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3028}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:29.311 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7249", 'reported_seq': 7329, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:27.484111+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:27.484111+0000', 'last_peered': '2024-09-15T21:08:27.484111+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:27.484111+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7249, 'log_dups_size': 0, 'ondisk_log_size': 7249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43022982, 'num_objects': 5252, 'num_object_clones': 0, 'num_object_copies': 21008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7248, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7146", 'reported_seq': 7203, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:25.366787+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:25.366787+0000', 'last_peered': '2024-09-15T21:08:25.366787+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:25.366787+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7146, 'log_dups_size': 0, 'ondisk_log_size': 7146, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43300108, 'num_objects': 5286, 'num_object_clones': 0, 'num_object_copies': 21144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7144, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 3007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7348", 'reported_seq': 7431, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:27.651077+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:27.651077+0000', 'last_peered': '2024-09-15T21:08:27.651077+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:27.651077+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7348, 'log_dups_size': 0, 'ondisk_log_size': 7348, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43685132, 'num_objects': 5333, 'num_object_clones': 0, 'num_object_copies': 21332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7346, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 3052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:29.312 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7254", 'reported_seq': 7323, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:24.807168+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:24.807168+0000', 'last_peered': '2024-09-15T21:08:24.807168+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:24.807168+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7254, 'log_dups_size': 0, 'ondisk_log_size': 7254, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43373836, 'num_objects': 5295, 'num_object_clones': 0, 'num_object_copies': 21180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5295, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7252, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:29.312 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:30.041 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:30.041 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33978/338100 objects degraded (10.050%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":375,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":84525,"data_bytes":692385582,"bytes_used":2805751808,"bytes_avail":383741304832,"bytes_total":386547056640,"degraded_objects":33978,"degraded_total":338100,"degraded_ratio":0.10049689440993789,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":60716,"recovering_keys_per_sec":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":233},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:30.078 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:30.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:30.660 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:30.661 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:30.687 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-15T21:08:30.031169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":688994094,"num_objects":84111,"num_object_clones":0,"num_object_copies":336444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110301,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116214,"ondisk_log_size":116214,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2742928,"kb_used_data":1268732,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374744432,"statfs":{"total":386547056640,"available":383738298368,"internally_reserved":0,"allocated":1299181568,"data_stored":1298521612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1509475890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,3,1,12,15,17,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23715840,"num_objects":-2895,"num_object_clones":0,"num_object_copies":-11580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2895,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":135,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008077"},"pg_stats":[{"pgid":"1.f","version":"33'7122","reported_seq":7168,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.433138+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:25.433138+0000","last_peered":"2024-09-15T21:08:25.433138+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:25.433138+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7122,"log_dups_size":0,"ondisk_log_size":7122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42844160,"num_objects":5230,"num_object_clones":0,"num_object_copies":20920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7122,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7339","reported_seq":9182,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.941556+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:29.941556+0000","last_peered":"2024-09-15T21:08:29.941556+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:29.941556+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42702092,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7215","reported_seq":7294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.618090+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:27.618090+0000","last_peered":"2024-09-15T21:08:27.618090+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:27.618090+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7215,"log_dups_size":0,"ondisk_log_size":7215,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42736286,"num_objects":5218,"num_object_clones":0,"num_object_copies":20872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5218,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7214,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2960}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7246","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.762251+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:29.762251+0000","last_peered":"2024-09-15T21:08:29.762251+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:29.762251+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42677516,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7306","reported_seq":7362,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.343902+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:25.343902+0000","last_peered":"2024-09-15T21:08:25.343902+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:25.343902+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43848972,"num_objects":5353,"num_object_clones":0,"num_object_copies":21412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":3042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7155","reported_seq":7200,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.350762+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:25.350762+0000","last_peered":"2024-09-15T21:08:25.350762+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:25.350762+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7155,"log_dups_size":0,"ondisk_log_size":7155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42770432,"num_objects":5221,"num_object_clones":0,"num_object_copies":20884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7155,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7481","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.798265+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:29.798265+0000","last_peered":"2024-09-15T21:08:29.798265+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:29.798265+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43341068,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7469","reported_seq":7538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.730639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:29.730639+0000","last_peered":"2024-09-15T21:08:29.730639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:29.730639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349260,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7237","reported_seq":7297,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.210717+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:25.210717+0000","last_peered":"2024-09-15T21:08:25.210717+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:25.210717+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42715672,"num_objects":5215,"num_object_clones":0,"num_object_copies":20860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7234,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2883}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7284","reported_seq":7365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.446996+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:27.446996+0000","last_peered":"2024-09-15T21:08:27.446996+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:27.446996+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7284,"log_dups_size":0,"ondisk_log_size":7284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43234572,"num_objects":5278,"num_object_clones":0,"num_object_copies":21112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7282,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2943}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7150","reported_seq":11813,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:25.292597+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:25.292597+0000","last_peered":"2024-09-15T21:08:25.292597+0000","last_clean":"2024-09-15T21:08:25.292597+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:25.292597+0000","last_undegraded":"2024-09-15T21:08:25.292597+0000","last_fullsized":"2024-09-15T21:08:25.292597+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7150,"log_dups_size":0,"ondisk_log_size":7150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42680320,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7143","reported_seq":7182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.321371+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:25.321371+0000","last_peered":"2024-09-15T21:08:25.321371+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:25.321371+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7143,"log_dups_size":0,"ondisk_log_size":7143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43285126,"num_objects":5284,"num_object_clones":0,"num_object_copies":21136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3028}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7249","reported_seq":7329,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.484111+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:27.484111+0000","last_peered":"2024-09-15T21:08:27.484111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:27.484111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43022982,"num_objects":5252,"num_object_clones":0,"num_object_copies":21008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7248,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7146","reported_seq":7203,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:25.366787+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:25.366787+0000","last_peered":"2024-09-15T21:08:25.366787+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:25.366787+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7146,"log_dups_size":0,"ondisk_log_size":7146,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43300108,"num_objects":5286,"num_object_clones":0,"num_object_copies":21144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7144,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":3007}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7348","reported_seq":7431,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:27.651077+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:27.651077+0000","last_peered":"2024-09-15T21:08:27.651077+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:27.651077+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7348,"log_dups_size":0,"ondisk_log_size":7348,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43685132,"num_objects":5333,"num_object_clones":0,"num_object_copies":21332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":3052}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7324","reported_seq":7393,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.843597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:29.843597+0000","last_peered":"2024-09-15T21:08:29.843597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:29.843597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":688994094,"num_objects":84111,"num_object_clones":0,"num_object_copies":336444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84111,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110301,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1296101376,"data_stored":1296101376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116214,"ondisk_log_size":116214,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736212,"kb_used_data":339392,"kb_used_omap":4,"kb_used_meta":396795,"kb_avail":93635628,"statfs":{"total":96636764160,"available":95882883072,"internally_reserved":0,"allocated":347537408,"data_stored":347372419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":406318800},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,10,11,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737192,"kb_used_data":336728,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93634648,"statfs":{"total":96636764160,"available":95881879552,"internally_reserved":0,"allocated":344809472,"data_stored":344644483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051639,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":542040,"kb_used_data":257736,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93829800,"statfs":{"total":96636764160,"available":96081715200,"internally_reserved":0,"allocated":263921664,"data_stored":263756675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727484,"kb_used_data":334876,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93644356,"statfs":{"total":96636764160,"available":95891820544,"internally_reserved":0,"allocated":342913024,"data_stored":342748035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341860352,"data_stored":341860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263618560,"data_stored":263618560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":344096768,"data_stored":344096768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346525696,"data_stored":346525696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:33.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:34.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:34.323 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:34.359 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-15T21:08:34.032661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681850670,"num_objects":83239,"num_object_clones":0,"num_object_copies":332956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111173,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117086,"ondisk_log_size":117086,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732648,"kb_used_data":1258440,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374754712,"statfs":{"total":386547056640,"available":383748825088,"internally_reserved":0,"allocated":1288642560,"data_stored":1287982604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1509475890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,6,1,5,12,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23429120,"num_objects":-2860,"num_object_clones":0,"num_object_copies":-11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007959"},"pg_stats":[{"pgid":"1.f","version":"33'7186","reported_seq":7232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.269404+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:30.269404+0000","last_peered":"2024-09-15T21:08:30.269404+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:30.269404+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7186,"log_dups_size":0,"ondisk_log_size":7186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42319872,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7339","reported_seq":9182,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.941556+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:29.941556+0000","last_peered":"2024-09-15T21:08:29.941556+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:29.941556+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42702092,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7286","reported_seq":7366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.468793+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:32.468793+0000","last_peered":"2024-09-15T21:08:32.468793+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:32.468793+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42154654,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7285,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7246","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.762251+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:29.762251+0000","last_peered":"2024-09-15T21:08:29.762251+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:29.762251+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42677516,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7394","reported_seq":7450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.351476+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:30.351476+0000","last_peered":"2024-09-15T21:08:30.351476+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:30.351476+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43128076,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7222","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.276015+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:30.276015+0000","last_peered":"2024-09-15T21:08:30.276015+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:30.276015+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7222,"log_dups_size":0,"ondisk_log_size":7222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42221568,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7481","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.798265+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:29.798265+0000","last_peered":"2024-09-15T21:08:29.798265+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:29.798265+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43341068,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7469","reported_seq":7538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.730639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:29.730639+0000","last_peered":"2024-09-15T21:08:29.730639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:29.730639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349260,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7334","reported_seq":7394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.540957+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:30.540957+0000","last_peered":"2024-09-15T21:08:30.540957+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:30.540957+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7334,"log_dups_size":0,"ondisk_log_size":7334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41921048,"num_objects":5118,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7331,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7361","reported_seq":7441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.659012+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:32.659012+0000","last_peered":"2024-09-15T21:08:32.659012+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:32.659012+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42603788,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7238","reported_seq":11898,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:30.517655+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:30.517655+0000","last_peered":"2024-09-15T21:08:30.517655+0000","last_clean":"2024-09-15T21:08:30.517655+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:30.517655+0000","last_undegraded":"2024-09-15T21:08:30.517655+0000","last_fullsized":"2024-09-15T21:08:30.517655+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41959424,"num_objects":5122,"num_object_clones":0,"num_object_copies":20488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1349,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7232","reported_seq":7272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.573747+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:30.573747+0000","last_peered":"2024-09-15T21:08:30.573747+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:30.573747+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556038,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7329","reported_seq":7409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.640008+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:32.640008+0000","last_peered":"2024-09-15T21:08:32.640008+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:32.640008+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7329,"log_dups_size":0,"ondisk_log_size":7329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42367622,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7225","reported_seq":7282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.307431+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:30.307431+0000","last_peered":"2024-09-15T21:08:30.307431+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:30.307431+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2928}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7420","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.587666+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:32.587666+0000","last_peered":"2024-09-15T21:08:32.587666+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:32.587666+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095308,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7324","reported_seq":7393,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.843597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:29.843597+0000","last_peered":"2024-09-15T21:08:29.843597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:29.843597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681850670,"num_objects":83239,"num_object_clones":0,"num_object_copies":332956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111173,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1285574656,"data_stored":1285574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117086,"ondisk_log_size":117086,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735256,"kb_used_data":334264,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93636584,"statfs":{"total":96636764160,"available":95883862016,"internally_reserved":0,"allocated":342286336,"data_stored":342121347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,1,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719792,"kb_used_data":331808,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93652048,"statfs":{"total":96636764160,"available":95899697152,"internally_reserved":0,"allocated":339771392,"data_stored":339606403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550116,"kb_used_data":257492,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93821724,"statfs":{"total":96636764160,"available":96073445376,"internally_reserved":0,"allocated":263671808,"data_stored":263506819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727484,"kb_used_data":334876,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93644356,"statfs":{"total":96636764160,"available":95891820544,"internally_reserved":0,"allocated":342913024,"data_stored":342748035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341860352,"data_stored":341860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263385088,"data_stored":263385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339030016,"data_stored":339030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341299200,"data_stored":341299200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:34.361 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:35.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:35.008 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:35.035 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-15T21:08:34.032661+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681850670,"num_objects":83239,"num_object_clones":0,"num_object_copies":332956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111173,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117086,"ondisk_log_size":117086,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732648,"kb_used_data":1258440,"kb_used_omap":14,"kb_used_meta":1474097,"kb_avail":374754712,"statfs":{"total":386547056640,"available":383748825088,"internally_reserved":0,"allocated":1288642560,"data_stored":1287982604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1509475890},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,0,6,1,5,12,10,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23429120,"num_objects":-2860,"num_object_clones":0,"num_object_copies":-11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2860,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":720896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007959"},"pg_stats":[{"pgid":"1.f","version":"33'7186","reported_seq":7232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.269404+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:30.269404+0000","last_peered":"2024-09-15T21:08:30.269404+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:30.269404+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7186,"log_dups_size":0,"ondisk_log_size":7186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42319872,"num_objects":5166,"num_object_clones":0,"num_object_copies":20664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7186,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2890}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7339","reported_seq":9182,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.941556+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:29.941556+0000","last_peered":"2024-09-15T21:08:29.941556+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:29.941556+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7339,"log_dups_size":0,"ondisk_log_size":7339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42702092,"num_objects":5213,"num_object_clones":0,"num_object_copies":20852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7337,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":891,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3720}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7286","reported_seq":7366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.468793+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:32.468793+0000","last_peered":"2024-09-15T21:08:32.468793+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:32.468793+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42154654,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7285,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7246","reported_seq":7314,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.762251+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:29.762251+0000","last_peered":"2024-09-15T21:08:29.762251+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:29.762251+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7246,"log_dups_size":0,"ondisk_log_size":7246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42677516,"num_objects":5210,"num_object_clones":0,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7244,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7394","reported_seq":7450,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.351476+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:30.351476+0000","last_peered":"2024-09-15T21:08:30.351476+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:30.351476+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43128076,"num_objects":5265,"num_object_clones":0,"num_object_copies":21060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2954}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7222","reported_seq":7267,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.276015+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:30.276015+0000","last_peered":"2024-09-15T21:08:30.276015+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:30.276015+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7222,"log_dups_size":0,"ondisk_log_size":7222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42221568,"num_objects":5154,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7222,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2838}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7481","reported_seq":7551,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.798265+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:29.798265+0000","last_peered":"2024-09-15T21:08:29.798265+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:29.798265+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7481,"log_dups_size":0,"ondisk_log_size":7481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43341068,"num_objects":5291,"num_object_clones":0,"num_object_copies":21164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7479,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2925}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7469","reported_seq":7538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.730639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:29.730639+0000","last_peered":"2024-09-15T21:08:29.730639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:29.730639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43349260,"num_objects":5292,"num_object_clones":0,"num_object_copies":21168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2818}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7334","reported_seq":7394,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.540957+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:30.540957+0000","last_peered":"2024-09-15T21:08:30.540957+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:30.540957+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7334,"log_dups_size":0,"ondisk_log_size":7334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41921048,"num_objects":5118,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7331,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7361","reported_seq":7441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.659012+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:32.659012+0000","last_peered":"2024-09-15T21:08:32.659012+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:32.659012+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42603788,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7238","reported_seq":11898,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:30.517655+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:30.517655+0000","last_peered":"2024-09-15T21:08:30.517655+0000","last_clean":"2024-09-15T21:08:30.517655+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:30.517655+0000","last_undegraded":"2024-09-15T21:08:30.517655+0000","last_fullsized":"2024-09-15T21:08:30.517655+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7238,"log_dups_size":0,"ondisk_log_size":7238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41959424,"num_objects":5122,"num_object_clones":0,"num_object_copies":20488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1349,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7232","reported_seq":7272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.573747+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:30.573747+0000","last_peered":"2024-09-15T21:08:30.573747+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:30.573747+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42556038,"num_objects":5195,"num_object_clones":0,"num_object_copies":20780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2939}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7329","reported_seq":7409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.640008+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:32.640008+0000","last_peered":"2024-09-15T21:08:32.640008+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:32.640008+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7329,"log_dups_size":0,"ondisk_log_size":7329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42367622,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7225","reported_seq":7282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:30.307431+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:30.307431+0000","last_peered":"2024-09-15T21:08:30.307431+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:30.307431+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7225,"log_dups_size":0,"ondisk_log_size":7225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42652940,"num_objects":5207,"num_object_clones":0,"num_object_copies":20828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2928}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7420","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.587666+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:32.587666+0000","last_peered":"2024-09-15T21:08:32.587666+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:32.587666+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095308,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7324","reported_seq":7393,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:29.843597+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:29.843597+0000","last_peered":"2024-09-15T21:08:29.843597+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:29.843597+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7324,"log_dups_size":0,"ondisk_log_size":7324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42800396,"num_objects":5225,"num_object_clones":0,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2910}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":681850670,"num_objects":83239,"num_object_clones":0,"num_object_copies":332956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":111173,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":913,"num_bytes_recovered":7266304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1285574656,"data_stored":1285574656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117086,"ondisk_log_size":117086,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735256,"kb_used_data":334264,"kb_used_omap":4,"kb_used_meta":400955,"kb_avail":93636584,"statfs":{"total":96636764160,"available":95883862016,"internally_reserved":0,"allocated":342286336,"data_stored":342121347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":410578640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,4,1,1,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719792,"kb_used_data":331808,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93652048,"statfs":{"total":96636764160,"available":95899697152,"internally_reserved":0,"allocated":339771392,"data_stored":339606403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550116,"kb_used_data":257492,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93821724,"statfs":{"total":96636764160,"available":96073445376,"internally_reserved":0,"allocated":263671808,"data_stored":263506819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247877,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727484,"kb_used_data":334876,"kb_used_omap":4,"kb_used_meta":392571,"kb_avail":93644356,"statfs":{"total":96636764160,"available":95891820544,"internally_reserved":0,"allocated":342913024,"data_stored":342748035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":401993422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":341860352,"data_stored":341860352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263385088,"data_stored":263385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339030016,"data_stored":339030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":341299200,"data_stored":341299200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:35.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:35.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7186", 'reported_seq': 7232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.269404+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:30.269404+0000', 'last_peered': '2024-09-15T21:08:30.269404+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:30.269404+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7186, 'log_dups_size': 0, 'ondisk_log_size': 7186, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42319872, 'num_objects': 5166, 'num_object_clones': 0, 'num_object_copies': 20664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7186, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2890}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:35.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7339", 'reported_seq': 9182, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:29.941556+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:29.941556+0000', 'last_peered': '2024-09-15T21:08:29.941556+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:29.941556+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7339, 'log_dups_size': 0, 'ondisk_log_size': 7339, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42702092, 'num_objects': 5213, 'num_object_clones': 0, 'num_object_copies': 20852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1493, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5213, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7337, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 891, 'num_bytes_recovered': 7266304, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3720}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7286", 'reported_seq': 7366, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:32.468793+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:32.468793+0000', 'last_peered': '2024-09-15T21:08:32.468793+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:32.468793+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7286, 'log_dups_size': 0, 'ondisk_log_size': 7286, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42154654, 'num_objects': 5147, 'num_object_clones': 0, 'num_object_copies': 20588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5147, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7285, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2889}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7246", 'reported_seq': 7314, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:29.762251+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:29.762251+0000', 'last_peered': '2024-09-15T21:08:29.762251+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:29.762251+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7246, 'log_dups_size': 0, 'ondisk_log_size': 7246, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42677516, 'num_objects': 5210, 'num_object_clones': 0, 'num_object_copies': 20840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7244, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2876}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7394", 'reported_seq': 7450, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.351476+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:30.351476+0000', 'last_peered': '2024-09-15T21:08:30.351476+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:30.351476+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7394, 'log_dups_size': 0, 'ondisk_log_size': 7394, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43128076, 'num_objects': 5265, 'num_object_clones': 0, 'num_object_copies': 21060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7392, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2954}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7222", 'reported_seq': 7267, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.276015+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:30.276015+0000', 'last_peered': '2024-09-15T21:08:30.276015+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:30.276015+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7222, 'log_dups_size': 0, 'ondisk_log_size': 7222, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42221568, 'num_objects': 5154, 'num_object_clones': 0, 'num_object_copies': 20616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7222, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2838}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7481", 'reported_seq': 7551, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:29.798265+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:29.798265+0000', 'last_peered': '2024-09-15T21:08:29.798265+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:29.798265+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7481, 'log_dups_size': 0, 'ondisk_log_size': 7481, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43341068, 'num_objects': 5291, 'num_object_clones': 0, 'num_object_copies': 21164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5291, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7479, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2925}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:35.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7469", 'reported_seq': 7538, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:29.730639+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:29.730639+0000', 'last_peered': '2024-09-15T21:08:29.730639+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:29.730639+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7469, 'log_dups_size': 0, 'ondisk_log_size': 7469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43349260, 'num_objects': 5292, 'num_object_clones': 0, 'num_object_copies': 21168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7467, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7334", 'reported_seq': 7394, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.540957+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:30.540957+0000', 'last_peered': '2024-09-15T21:08:30.540957+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:30.540957+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7334, 'log_dups_size': 0, 'ondisk_log_size': 7334, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41921048, 'num_objects': 5118, 'num_object_clones': 0, 'num_object_copies': 20472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5118, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7331, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2786}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7361", 'reported_seq': 7441, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:32.659012+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:32.659012+0000', 'last_peered': '2024-09-15T21:08:32.659012+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:32.659012+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7361, 'log_dups_size': 0, 'ondisk_log_size': 7361, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42603788, 'num_objects': 5201, 'num_object_clones': 0, 'num_object_copies': 20804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5201, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7359, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2866}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7232", 'reported_seq': 7272, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.573747+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:30.573747+0000', 'last_peered': '2024-09-15T21:08:30.573747+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:30.573747+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7232, 'log_dups_size': 0, 'ondisk_log_size': 7232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42556038, 'num_objects': 5195, 'num_object_clones': 0, 'num_object_copies': 20780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7231, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2939}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7329", 'reported_seq': 7409, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:32.640008+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:32.640008+0000', 'last_peered': '2024-09-15T21:08:32.640008+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:32.640008+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7329, 'log_dups_size': 0, 'ondisk_log_size': 7329, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42367622, 'num_objects': 5172, 'num_object_clones': 0, 'num_object_copies': 20688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7328, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2886}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:35.038 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7225", 'reported_seq': 7282, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:30.307431+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:30.307431+0000', 'last_peered': '2024-09-15T21:08:30.307431+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:30.307431+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7225, 'log_dups_size': 0, 'ondisk_log_size': 7225, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42652940, 'num_objects': 5207, 'num_object_clones': 0, 'num_object_copies': 20828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7223, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:35.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:35.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7420", 'reported_seq': 7503, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:32.587666+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:32.587666+0000', 'last_peered': '2024-09-15T21:08:32.587666+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:32.587666+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7420, 'log_dups_size': 0, 'ondisk_log_size': 7420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 43095308, 'num_objects': 5261, 'num_object_clones': 0, 'num_object_copies': 21044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5261, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7418, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2980}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:35.039 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:35.039 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7324", 'reported_seq': 7393, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:29.843597+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:29.843597+0000', 'last_peered': '2024-09-15T21:08:29.843597+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:29.843597+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7324, 'log_dups_size': 0, 'ondisk_log_size': 7324, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42800396, 'num_objects': 5225, 'num_object_clones': 0, 'num_object_copies': 20900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5225, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7322, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:35.039 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:35.788 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:35.788 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33934/332956 objects degraded (10.192%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":381,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":83239,"data_bytes":681850670,"bytes_used":2798231552,"bytes_avail":383748825088,"bytes_total":386547056640,"degraded_objects":33934,"degraded_total":332956,"degraded_ratio":0.10191737046336452,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":60034,"recovering_keys_per_sec":0,"num_objects_recovered":90,"num_bytes_recovered":720896,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":238},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:35.817 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:35.817 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:36.418 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:36.419 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:36.446 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-15T21:08:36.033565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":674002734,"num_objects":82281,"num_object_clones":0,"num_object_copies":329124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112131,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118044,"ondisk_log_size":118044,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2714336,"kb_used_data":1248480,"kb_used_omap":14,"kb_used_meta":1465777,"kb_avail":374773024,"statfs":{"total":386547056640,"available":383767576576,"internally_reserved":0,"allocated":1278443520,"data_stored":1277783564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1500956210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,10,8,6,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28524544,"num_objects":-3482,"num_object_clones":0,"num_object_copies":-13928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3482,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":1064960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008232"},"pg_stats":[{"pgid":"1.f","version":"33'7250","reported_seq":7296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.546241+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:35.546241+0000","last_peered":"2024-09-15T21:08:35.546241+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:35.546241+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7250,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7426","reported_seq":9357,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.058065+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:35.058065+0000","last_peered":"2024-09-15T21:08:35.058065+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:35.058065+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41989388,"num_objects":5126,"num_object_clones":0,"num_object_copies":20504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7286","reported_seq":7366,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.468793+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:32.468793+0000","last_peered":"2024-09-15T21:08:32.468793+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:32.468793+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7286,"log_dups_size":0,"ondisk_log_size":7286,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42154654,"num_objects":5147,"num_object_clones":0,"num_object_copies":20588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5147,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7285,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2889}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7332","reported_seq":7401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.944047+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:34.944047+0000","last_peered":"2024-09-15T21:08:34.944047+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:34.944047+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7332,"log_dups_size":0,"ondisk_log_size":7332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7468","reported_seq":7524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.437479+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:35.437479+0000","last_peered":"2024-09-15T21:08:35.437479+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:35.437479+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521868,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7466,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7314","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.551827+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:35.551827+0000","last_peered":"2024-09-15T21:08:35.551827+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:35.551827+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41467904,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7314,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7568","reported_seq":7638,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.020167+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:35.020167+0000","last_peered":"2024-09-15T21:08:35.020167+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:35.020167+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628364,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7547","reported_seq":7616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.968069+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:34.968069+0000","last_peered":"2024-09-15T21:08:34.968069+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:34.968069+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7545,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7401","reported_seq":7461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.435742+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:35.435742+0000","last_peered":"2024-09-15T21:08:35.435742+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:35.435742+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41372184,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7398,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7361","reported_seq":7441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.659012+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:32.659012+0000","last_peered":"2024-09-15T21:08:32.659012+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:32.659012+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42603788,"num_objects":5201,"num_object_clones":0,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7359,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2866}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7306","reported_seq":11968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:35.556709+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:35.556709+0000","last_peered":"2024-09-15T21:08:35.556709+0000","last_clean":"2024-09-15T21:08:35.556709+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:35.556709+0000","last_undegraded":"2024-09-15T21:08:35.556709+0000","last_fullsized":"2024-09-15T21:08:35.556709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7311","reported_seq":7350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.512170+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:35.512170+0000","last_peered":"2024-09-15T21:08:35.512170+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:35.512170+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7311,"log_dups_size":0,"ondisk_log_size":7311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41908870,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7329","reported_seq":7409,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.640008+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:32.640008+0000","last_peered":"2024-09-15T21:08:32.640008+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:32.640008+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7329,"log_dups_size":0,"ondisk_log_size":7329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42367622,"num_objects":5172,"num_object_clones":0,"num_object_copies":20688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7328,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2886}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7313","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.510546+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:35.510546+0000","last_peered":"2024-09-15T21:08:35.510546+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:35.510546+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7420","reported_seq":7503,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:32.587666+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:32.587666+0000","last_peered":"2024-09-15T21:08:32.587666+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:32.587666+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7420,"log_dups_size":0,"ondisk_log_size":7420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43095308,"num_objects":5261,"num_object_clones":0,"num_object_copies":21044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7418,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2980}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7412","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.009177+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:35.009177+0000","last_peered":"2024-09-15T21:08:35.009177+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:35.009177+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079500,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7410,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":674002734,"num_objects":82281,"num_object_clones":0,"num_object_copies":329124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82281,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112131,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1275150336,"data_stored":1275150336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118044,"ondisk_log_size":118044,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717852,"kb_used_data":329356,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93653988,"statfs":{"total":96636764160,"available":95901683712,"internally_reserved":0,"allocated":337260544,"data_stored":337095555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,1,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719792,"kb_used_data":331808,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93652048,"statfs":{"total":96636764160,"available":95899697152,"internally_reserved":0,"allocated":339771392,"data_stored":339606403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051640,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":550116,"kb_used_data":257492,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93821724,"statfs":{"total":96636764160,"available":96073445376,"internally_reserved":0,"allocated":263671808,"data_stored":263506819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726576,"kb_used_data":329824,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645264,"statfs":{"total":96636764160,"available":95892750336,"internally_reserved":0,"allocated":337739776,"data_stored":337574787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336637952,"data_stored":336637952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263385088,"data_stored":263385088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339030016,"data_stored":339030016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336097280,"data_stored":336097280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:39.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:40.090 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:40.090 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:40.131 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-15T21:08:40.035175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671659822,"num_objects":81995,"num_object_clones":0,"num_object_copies":327980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112417,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118330,"ondisk_log_size":118330,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739780,"kb_used_data":1243396,"kb_used_omap":14,"kb_used_meta":1496305,"kb_avail":374747580,"statfs":{"total":386547056640,"available":383741521920,"internally_reserved":0,"allocated":1273237504,"data_stored":1272577548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1532216882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,3,13,11,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004690"},"pg_stats":[{"pgid":"1.f","version":"33'7250","reported_seq":7296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.546241+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:35.546241+0000","last_peered":"2024-09-15T21:08:35.546241+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:35.546241+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7250,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7426","reported_seq":9357,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.058065+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:35.058065+0000","last_peered":"2024-09-15T21:08:35.058065+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:35.058065+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41989388,"num_objects":5126,"num_object_clones":0,"num_object_copies":20504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7358","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.420278+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:37.420278+0000","last_peered":"2024-09-15T21:08:37.420278+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:37.420278+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7358,"log_dups_size":0,"ondisk_log_size":7358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41564830,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7357,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7332","reported_seq":7401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.944047+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:34.944047+0000","last_peered":"2024-09-15T21:08:34.944047+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:34.944047+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7332,"log_dups_size":0,"ondisk_log_size":7332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7468","reported_seq":7524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.437479+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:35.437479+0000","last_peered":"2024-09-15T21:08:35.437479+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:35.437479+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521868,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7466,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7314","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.551827+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:35.551827+0000","last_peered":"2024-09-15T21:08:35.551827+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:35.551827+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41467904,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7314,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7568","reported_seq":7638,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.020167+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:35.020167+0000","last_peered":"2024-09-15T21:08:35.020167+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:35.020167+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628364,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7547","reported_seq":7616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.968069+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:34.968069+0000","last_peered":"2024-09-15T21:08:34.968069+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:34.968069+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7545,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7401","reported_seq":7461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.435742+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:35.435742+0000","last_peered":"2024-09-15T21:08:35.435742+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:35.435742+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41372184,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7398,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7429","reported_seq":7510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.602836+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:37.602836+0000","last_peered":"2024-09-15T21:08:37.602836+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:37.602836+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42046732,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7306","reported_seq":11968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:35.556709+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:35.556709+0000","last_peered":"2024-09-15T21:08:35.556709+0000","last_clean":"2024-09-15T21:08:35.556709+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:35.556709+0000","last_undegraded":"2024-09-15T21:08:35.556709+0000","last_fullsized":"2024-09-15T21:08:35.556709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7311","reported_seq":7350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.512170+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:35.512170+0000","last_peered":"2024-09-15T21:08:35.512170+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:35.512170+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7311,"log_dups_size":0,"ondisk_log_size":7311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41908870,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7400","reported_seq":7480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.670414+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:37.670414+0000","last_peered":"2024-09-15T21:08:37.670414+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:37.670414+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41785990,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7313","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.510546+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:35.510546+0000","last_peered":"2024-09-15T21:08:35.510546+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:35.510546+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7495","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.696280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:37.696280+0000","last_peered":"2024-09-15T21:08:37.696280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:37.696280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7495,"log_dups_size":0,"ondisk_log_size":7495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480908,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7412","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.009177+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:35.009177+0000","last_peered":"2024-09-15T21:08:35.009177+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:35.009177+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079500,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7410,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":671659822,"num_objects":81995,"num_object_clones":0,"num_object_copies":327980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112417,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269800960,"data_stored":1269800960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118330,"ondisk_log_size":118330,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717852,"kb_used_data":329356,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93653988,"statfs":{"total":96636764160,"available":95901683712,"internally_reserved":0,"allocated":337260544,"data_stored":337095555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,1,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741304,"kb_used_data":326952,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93630536,"statfs":{"total":96636764160,"available":95877668864,"internally_reserved":0,"allocated":334798848,"data_stored":334633859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554048,"kb_used_data":257264,"kb_used_omap":4,"kb_used_meta":296763,"kb_avail":93817792,"statfs":{"total":96636764160,"available":96069419008,"internally_reserved":0,"allocated":263438336,"data_stored":263273347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":303886031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726576,"kb_used_data":329824,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645264,"statfs":{"total":96636764160,"available":95892750336,"internally_reserved":0,"allocated":337739776,"data_stored":337574787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336637952,"data_stored":336637952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263143424,"data_stored":263143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333922304,"data_stored":333922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336097280,"data_stored":336097280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:40.132 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:40.756 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:40.756 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:40.792 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-15T21:08:40.035175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":671659822,"num_objects":81995,"num_object_clones":0,"num_object_copies":327980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112417,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118330,"ondisk_log_size":118330,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2739780,"kb_used_data":1243396,"kb_used_omap":14,"kb_used_meta":1496305,"kb_avail":374747580,"statfs":{"total":386547056640,"available":383741521920,"internally_reserved":0,"allocated":1273237504,"data_stored":1272577548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1532216882},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,1,3,13,11,10,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20725760,"num_objects":-2530,"num_object_clones":0,"num_object_copies":-10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-86,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2530,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":704512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004690"},"pg_stats":[{"pgid":"1.f","version":"33'7250","reported_seq":7296,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.546241+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:35.546241+0000","last_peered":"2024-09-15T21:08:35.546241+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:35.546241+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7250,"log_dups_size":0,"ondisk_log_size":7250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41795584,"num_objects":5102,"num_object_clones":0,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7250,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7426","reported_seq":9357,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.058065+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:35.058065+0000","last_peered":"2024-09-15T21:08:35.058065+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:35.058065+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41989388,"num_objects":5126,"num_object_clones":0,"num_object_copies":20504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1451,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7424,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":933,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3675}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7358","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.420278+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:37.420278+0000","last_peered":"2024-09-15T21:08:37.420278+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:37.420278+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7358,"log_dups_size":0,"ondisk_log_size":7358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41564830,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7357,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7332","reported_seq":7401,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.944047+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:34.944047+0000","last_peered":"2024-09-15T21:08:34.944047+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:34.944047+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7332,"log_dups_size":0,"ondisk_log_size":7332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41973004,"num_objects":5124,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7330,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2790}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7468","reported_seq":7524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.437479+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:35.437479+0000","last_peered":"2024-09-15T21:08:35.437479+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:35.437479+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7468,"log_dups_size":0,"ondisk_log_size":7468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42521868,"num_objects":5191,"num_object_clones":0,"num_object_copies":20764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7466,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7314","reported_seq":7358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.551827+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:35.551827+0000","last_peered":"2024-09-15T21:08:35.551827+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:35.551827+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7314,"log_dups_size":0,"ondisk_log_size":7314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41467904,"num_objects":5062,"num_object_clones":0,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7314,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2746}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7568","reported_seq":7638,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.020167+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:35.020167+0000","last_peered":"2024-09-15T21:08:35.020167+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:35.020167+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7568,"log_dups_size":0,"ondisk_log_size":7568,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42628364,"num_objects":5204,"num_object_clones":0,"num_object_copies":20816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7566,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2838}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7547","reported_seq":7616,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:34.968069+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:34.968069+0000","last_peered":"2024-09-15T21:08:34.968069+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:34.968069+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42710284,"num_objects":5214,"num_object_clones":0,"num_object_copies":20856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7545,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2740}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7401","reported_seq":7461,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.435742+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:35.435742+0000","last_peered":"2024-09-15T21:08:35.435742+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:35.435742+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41372184,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7398,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7429","reported_seq":7510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.602836+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:37.602836+0000","last_peered":"2024-09-15T21:08:37.602836+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:37.602836+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42046732,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7306","reported_seq":11968,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:35.556709+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:35.556709+0000","last_peered":"2024-09-15T21:08:35.556709+0000","last_clean":"2024-09-15T21:08:35.556709+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:35.556709+0000","last_undegraded":"2024-09-15T21:08:35.556709+0000","last_fullsized":"2024-09-15T21:08:35.556709+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7306,"log_dups_size":0,"ondisk_log_size":7306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41402368,"num_objects":5054,"num_object_clones":0,"num_object_copies":20216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7311","reported_seq":7350,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.512170+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:35.512170+0000","last_peered":"2024-09-15T21:08:35.512170+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:35.512170+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7311,"log_dups_size":0,"ondisk_log_size":7311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41908870,"num_objects":5116,"num_object_clones":0,"num_object_copies":20464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7310,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2860}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7400","reported_seq":7480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.670414+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:37.670414+0000","last_peered":"2024-09-15T21:08:37.670414+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:37.670414+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41785990,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7313","reported_seq":7370,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.510546+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:35.510546+0000","last_peered":"2024-09-15T21:08:35.510546+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:35.510546+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7313,"log_dups_size":0,"ondisk_log_size":7313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41932044,"num_objects":5119,"num_object_clones":0,"num_object_copies":20476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2840}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7495","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.696280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:37.696280+0000","last_peered":"2024-09-15T21:08:37.696280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:37.696280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7495,"log_dups_size":0,"ondisk_log_size":7495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480908,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7412","reported_seq":7481,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:35.009177+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:35.009177+0000","last_peered":"2024-09-15T21:08:35.009177+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:35.009177+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7412,"log_dups_size":0,"ondisk_log_size":7412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42079500,"num_objects":5137,"num_object_clones":0,"num_object_copies":20548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7410,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2822}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":671659822,"num_objects":81995,"num_object_clones":0,"num_object_copies":327980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33892,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81995,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":112417,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":7610368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1269800960,"data_stored":1269800960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118330,"ondisk_log_size":118330,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717852,"kb_used_data":329356,"kb_used_omap":4,"kb_used_meta":388475,"kb_avail":93653988,"statfs":{"total":96636764160,"available":95901683712,"internally_reserved":0,"allocated":337260544,"data_stored":337095555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":397799120},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,4,1,3,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741304,"kb_used_data":326952,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93630536,"statfs":{"total":96636764160,"available":95877668864,"internally_reserved":0,"allocated":334798848,"data_stored":334633859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554048,"kb_used_data":257264,"kb_used_omap":4,"kb_used_meta":296763,"kb_avail":93817792,"statfs":{"total":96636764160,"available":96069419008,"internally_reserved":0,"allocated":263438336,"data_stored":263273347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":303886031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247878,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726576,"kb_used_data":329824,"kb_used_omap":4,"kb_used_meta":396731,"kb_avail":93645264,"statfs":{"total":96636764160,"available":95892750336,"internally_reserved":0,"allocated":337739776,"data_stored":337574787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":406253262},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,0,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":336637952,"data_stored":336637952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263143424,"data_stored":263143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333922304,"data_stored":333922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":336097280,"data_stored":336097280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:40.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:40.793 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7250", 'reported_seq': 7296, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.546241+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:35.546241+0000', 'last_peered': '2024-09-15T21:08:35.546241+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:35.546241+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7250, 'log_dups_size': 0, 'ondisk_log_size': 7250, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41795584, 'num_objects': 5102, 'num_object_clones': 0, 'num_object_copies': 20408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5102, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7250, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.793 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:40.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7426", 'reported_seq': 9357, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.058065+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:35.058065+0000', 'last_peered': '2024-09-15T21:08:35.058065+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:35.058065+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7426, 'log_dups_size': 0, 'ondisk_log_size': 7426, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41989388, 'num_objects': 5126, 'num_object_clones': 0, 'num_object_copies': 20504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1451, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7424, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 933, 'num_bytes_recovered': 7610368, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3675}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:40.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:40.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7358", 'reported_seq': 7438, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:37.420278+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:37.420278+0000', 'last_peered': '2024-09-15T21:08:37.420278+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:37.420278+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7358, 'log_dups_size': 0, 'ondisk_log_size': 7358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41564830, 'num_objects': 5075, 'num_object_clones': 0, 'num_object_copies': 20300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5075, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7357, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2817}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:40.794 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:40.794 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7332", 'reported_seq': 7401, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:34.944047+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:34.944047+0000', 'last_peered': '2024-09-15T21:08:34.944047+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:34.944047+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7332, 'log_dups_size': 0, 'ondisk_log_size': 7332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41973004, 'num_objects': 5124, 'num_object_clones': 0, 'num_object_copies': 20496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5124, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7330, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2790}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7468", 'reported_seq': 7524, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.437479+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:35.437479+0000', 'last_peered': '2024-09-15T21:08:35.437479+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:35.437479+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7468, 'log_dups_size': 0, 'ondisk_log_size': 7468, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42521868, 'num_objects': 5191, 'num_object_clones': 0, 'num_object_copies': 20764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7466, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2880}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7314", 'reported_seq': 7358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.551827+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:35.551827+0000', 'last_peered': '2024-09-15T21:08:35.551827+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:35.551827+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7314, 'log_dups_size': 0, 'ondisk_log_size': 7314, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41467904, 'num_objects': 5062, 'num_object_clones': 0, 'num_object_copies': 20248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7314, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2746}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7568", 'reported_seq': 7638, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.020167+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:35.020167+0000', 'last_peered': '2024-09-15T21:08:35.020167+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:35.020167+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7568, 'log_dups_size': 0, 'ondisk_log_size': 7568, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42628364, 'num_objects': 5204, 'num_object_clones': 0, 'num_object_copies': 20816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5204, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7566, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2838}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7547", 'reported_seq': 7616, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:34.968069+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:34.968069+0000', 'last_peered': '2024-09-15T21:08:34.968069+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:34.968069+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7547, 'log_dups_size': 0, 'ondisk_log_size': 7547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42710284, 'num_objects': 5214, 'num_object_clones': 0, 'num_object_copies': 20856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7545, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2740}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:40.795 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7401", 'reported_seq': 7461, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.435742+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:35.435742+0000', 'last_peered': '2024-09-15T21:08:35.435742+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:35.435742+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7401, 'log_dups_size': 0, 'ondisk_log_size': 7401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41372184, 'num_objects': 5051, 'num_object_clones': 0, 'num_object_copies': 20204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7398, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2719}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7429", 'reported_seq': 7510, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:37.602836+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:37.602836+0000', 'last_peered': '2024-09-15T21:08:37.602836+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:37.602836+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7429, 'log_dups_size': 0, 'ondisk_log_size': 7429, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42046732, 'num_objects': 5133, 'num_object_clones': 0, 'num_object_copies': 20532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7427, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2798}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7311", 'reported_seq': 7350, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.512170+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:35.512170+0000', 'last_peered': '2024-09-15T21:08:35.512170+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:35.512170+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7311, 'log_dups_size': 0, 'ondisk_log_size': 7311, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41908870, 'num_objects': 5116, 'num_object_clones': 0, 'num_object_copies': 20464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5116, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7310, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2860}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:40.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7400", 'reported_seq': 7480, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:37.670414+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:37.670414+0000', 'last_peered': '2024-09-15T21:08:37.670414+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:37.670414+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7400, 'log_dups_size': 0, 'ondisk_log_size': 7400, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41785990, 'num_objects': 5101, 'num_object_clones': 0, 'num_object_copies': 20404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5101, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7399, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2815}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7313", 'reported_seq': 7370, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.510546+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:35.510546+0000', 'last_peered': '2024-09-15T21:08:35.510546+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:35.510546+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7313, 'log_dups_size': 0, 'ondisk_log_size': 7313, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41932044, 'num_objects': 5119, 'num_object_clones': 0, 'num_object_copies': 20476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7311, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2840}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7495", 'reported_seq': 7578, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:37.696280+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:37.696280+0000', 'last_peered': '2024-09-15T21:08:37.696280+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:37.696280+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7495, 'log_dups_size': 0, 'ondisk_log_size': 7495, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42480908, 'num_objects': 5186, 'num_object_clones': 0, 'num_object_copies': 20744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7493, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2905}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:40.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7412", 'reported_seq': 7481, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:35.009177+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:35.009177+0000', 'last_peered': '2024-09-15T21:08:35.009177+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:35.009177+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7412, 'log_dups_size': 0, 'ondisk_log_size': 7412, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 42079500, 'num_objects': 5137, 'num_object_clones': 0, 'num_object_copies': 20548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5137, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7410, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:40.798 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:41.557 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:41.557 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33892/327980 objects degraded (10.334%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":387,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":81995,"data_bytes":671659822,"bytes_used":2805534720,"bytes_avail":383741521920,"bytes_total":386547056640,"degraded_objects":33892,"degraded_total":327980,"degraded_ratio":0.10333556924202697,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":58686,"recovering_keys_per_sec":0,"num_objects_recovered":88,"num_bytes_recovered":704512,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":210},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:41.581 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:41.581 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:42.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:42.184 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:42.207 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-15T21:08:42.035803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663664430,"num_objects":81019,"num_object_clones":0,"num_object_copies":324076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113393,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119306,"ondisk_log_size":119306,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2774204,"kb_used_data":1233404,"kb_used_omap":14,"kb_used_meta":1540721,"kb_avail":374713156,"statfs":{"total":386547056640,"available":383706271744,"internally_reserved":0,"allocated":1263005696,"data_stored":1262345740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1577698866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,3,4,12,16,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25329664,"num_objects":-3092,"num_object_clones":0,"num_object_copies":-12368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3092,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004634"},"pg_stats":[{"pgid":"1.f","version":"33'7322","reported_seq":7368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.533174+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:40.533174+0000","last_peered":"2024-09-15T21:08:40.533174+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:40.533174+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7322,"log_dups_size":0,"ondisk_log_size":7322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41205760,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7511","reported_seq":9537,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.069756+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:40.069756+0000","last_peered":"2024-09-15T21:08:40.069756+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:40.069756+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41293068,"num_objects":5041,"num_object_clones":0,"num_object_copies":20164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":981,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7358","reported_seq":7438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.420278+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:37.420278+0000","last_peered":"2024-09-15T21:08:37.420278+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:37.420278+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7358,"log_dups_size":0,"ondisk_log_size":7358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41564830,"num_objects":5075,"num_object_clones":0,"num_object_copies":20300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5075,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7357,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2817}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7418","reported_seq":7487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.068094+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:40.068094+0000","last_peered":"2024-09-15T21:08:40.068094+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:40.068094+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7418,"log_dups_size":0,"ondisk_log_size":7418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268492,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7549","reported_seq":7605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.513399+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:40.513399+0000","last_peered":"2024-09-15T21:08:40.513399+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:40.513399+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41858316,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7413","reported_seq":7458,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.477893+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:40.477893+0000","last_peered":"2024-09-15T21:08:40.477893+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:40.477893+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40656896,"num_objects":4963,"num_object_clones":0,"num_object_copies":19852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7413,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7655","reported_seq":7723,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.073571+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:40.073571+0000","last_peered":"2024-09-15T21:08:40.073571+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:40.073571+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7655,"log_dups_size":0,"ondisk_log_size":7655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41915660,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7653,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7633","reported_seq":7702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.042612+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:40.042612+0000","last_peered":"2024-09-15T21:08:40.042612+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:40.042612+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005772,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7478","reported_seq":7538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.588262+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:40.588262+0000","last_peered":"2024-09-15T21:08:40.588262+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:40.588262+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40741400,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7475,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7429","reported_seq":7510,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.602836+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:37.602836+0000","last_peered":"2024-09-15T21:08:37.602836+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:37.602836+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7429,"log_dups_size":0,"ondisk_log_size":7429,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42046732,"num_objects":5133,"num_object_clones":0,"num_object_copies":20532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7427,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2798}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7379","reported_seq":12042,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:40.400201+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:40.400201+0000","last_peered":"2024-09-15T21:08:40.400201+0000","last_clean":"2024-09-15T21:08:40.400201+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:40.400201+0000","last_undegraded":"2024-09-15T21:08:40.400201+0000","last_fullsized":"2024-09-15T21:08:40.400201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7379,"log_dups_size":0,"ondisk_log_size":7379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40804352,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7384","reported_seq":7424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.545212+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:40.545212+0000","last_peered":"2024-09-15T21:08:40.545212+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:40.545212+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41310854,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7400","reported_seq":7480,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.670414+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:37.670414+0000","last_peered":"2024-09-15T21:08:37.670414+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:37.670414+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41785990,"num_objects":5101,"num_object_clones":0,"num_object_copies":20404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7399,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2815}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7397","reported_seq":7454,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.458596+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:40.458596+0000","last_peered":"2024-09-15T21:08:40.458596+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:40.458596+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243916,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7495","reported_seq":7578,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:37.696280+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:37.696280+0000","last_peered":"2024-09-15T21:08:37.696280+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:37.696280+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7495,"log_dups_size":0,"ondisk_log_size":7495,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42480908,"num_objects":5186,"num_object_clones":0,"num_object_copies":20744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2905}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7485","reported_seq":7554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:39.715970+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:39.715970+0000","last_peered":"2024-09-15T21:08:39.715970+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:39.715970+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7485,"log_dups_size":0,"ondisk_log_size":7485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41481484,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":663664430,"num_objects":81019,"num_object_clones":0,"num_object_copies":324076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113393,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1259524096,"data_stored":1259524096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119306,"ondisk_log_size":119306,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735112,"kb_used_data":324408,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93636728,"statfs":{"total":96636764160,"available":95884009472,"internally_reserved":0,"allocated":332193792,"data_stored":332028803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":420540112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741304,"kb_used_data":326952,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93630536,"statfs":{"total":96636764160,"available":95877668864,"internally_reserved":0,"allocated":334798848,"data_stored":334633859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,1,0,3,4,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051641,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554048,"kb_used_data":257264,"kb_used_omap":4,"kb_used_meta":296763,"kb_avail":93817792,"statfs":{"total":96636764160,"available":96069419008,"internally_reserved":0,"allocated":263438336,"data_stored":263273347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":303886031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743740,"kb_used_data":324780,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93628100,"statfs":{"total":96636764160,"available":95875174400,"internally_reserved":0,"allocated":332574720,"data_stored":332409731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331501568,"data_stored":331501568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":263143424,"data_stored":263143424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":333922304,"data_stored":333922304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330956800,"data_stored":330956800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:45.209 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:45.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:45.793 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:45.820 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-15T21:08:44.036561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661485358,"num_objects":80753,"num_object_clones":0,"num_object_copies":323012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113659,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119572,"ondisk_log_size":119572,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760756,"kb_used_data":1228276,"kb_used_omap":14,"kb_used_meta":1532401,"kb_avail":374726604,"statfs":{"total":386547056640,"available":383720042496,"internally_reserved":0,"allocated":1257754624,"data_stored":1257094668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1569179186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,7,9,12,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22822912,"num_objects":-2786,"num_object_clones":0,"num_object_copies":-11144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004548"},"pg_stats":[{"pgid":"1.f","version":"33'7322","reported_seq":7368,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.533174+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:40.533174+0000","last_peered":"2024-09-15T21:08:40.533174+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:40.533174+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7322,"log_dups_size":0,"ondisk_log_size":7322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41205760,"num_objects":5030,"num_object_clones":0,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7322,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2754}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7511","reported_seq":9537,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.069756+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:40.069756+0000","last_peered":"2024-09-15T21:08:40.069756+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:40.069756+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7511,"log_dups_size":0,"ondisk_log_size":7511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41293068,"num_objects":5041,"num_object_clones":0,"num_object_copies":20164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7509,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":981,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3638}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7427","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.672893+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:42.672893+0000","last_peered":"2024-09-15T21:08:42.672893+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:42.672893+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40999582,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7426,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7418","reported_seq":7487,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.068094+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:40.068094+0000","last_peered":"2024-09-15T21:08:40.068094+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:40.068094+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7418,"log_dups_size":0,"ondisk_log_size":7418,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41268492,"num_objects":5038,"num_object_clones":0,"num_object_copies":20152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7549","reported_seq":7605,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.513399+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:40.513399+0000","last_peered":"2024-09-15T21:08:40.513399+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:40.513399+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7549,"log_dups_size":0,"ondisk_log_size":7549,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41858316,"num_objects":5110,"num_object_clones":0,"num_object_copies":20440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2799}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7413","reported_seq":7458,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.477893+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:40.477893+0000","last_peered":"2024-09-15T21:08:40.477893+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:40.477893+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7413,"log_dups_size":0,"ondisk_log_size":7413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40656896,"num_objects":4963,"num_object_clones":0,"num_object_copies":19852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7413,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7655","reported_seq":7723,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.073571+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:40.073571+0000","last_peered":"2024-09-15T21:08:40.073571+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:40.073571+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7655,"log_dups_size":0,"ondisk_log_size":7655,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41915660,"num_objects":5117,"num_object_clones":0,"num_object_copies":20468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7653,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2751}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7633","reported_seq":7702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.042612+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:40.042612+0000","last_peered":"2024-09-15T21:08:40.042612+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:40.042612+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42005772,"num_objects":5128,"num_object_clones":0,"num_object_copies":20512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2654}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7478","reported_seq":7538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.588262+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:40.588262+0000","last_peered":"2024-09-15T21:08:40.588262+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:40.588262+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40741400,"num_objects":4974,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7475,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2642}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7499","reported_seq":7580,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.686389+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:42.686389+0000","last_peered":"2024-09-15T21:08:42.686389+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:42.686389+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473292,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2728}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7379","reported_seq":12042,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:40.400201+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:40.400201+0000","last_peered":"2024-09-15T21:08:40.400201+0000","last_clean":"2024-09-15T21:08:40.400201+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:40.400201+0000","last_undegraded":"2024-09-15T21:08:40.400201+0000","last_fullsized":"2024-09-15T21:08:40.400201+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7379,"log_dups_size":0,"ondisk_log_size":7379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40804352,"num_objects":4981,"num_object_clones":0,"num_object_copies":19924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1490,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7384","reported_seq":7424,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.545212+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:40.545212+0000","last_peered":"2024-09-15T21:08:40.545212+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:40.545212+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7384,"log_dups_size":0,"ondisk_log_size":7384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41310854,"num_objects":5043,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7383,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2787}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7455","reported_seq":7535,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.523864+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:42.523864+0000","last_peered":"2024-09-15T21:08:42.523864+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:42.523864+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41335430,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7454,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7397","reported_seq":7454,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:40.458596+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:40.458596+0000","last_peered":"2024-09-15T21:08:40.458596+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:40.458596+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7397,"log_dups_size":0,"ondisk_log_size":7397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41243916,"num_objects":5035,"num_object_clones":0,"num_object_copies":20140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7395,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2756}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7567","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.747508+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:42.747508+0000","last_peered":"2024-09-15T21:08:42.747508+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:42.747508+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41891084,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7485","reported_seq":7554,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:39.715970+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:39.715970+0000","last_peered":"2024-09-15T21:08:39.715970+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:39.715970+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7485,"log_dups_size":0,"ondisk_log_size":7485,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41481484,"num_objects":5064,"num_object_clones":0,"num_object_copies":20256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7483,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2749}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":661485358,"num_objects":80753,"num_object_clones":0,"num_object_copies":323012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33844,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80753,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113659,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1003,"num_bytes_recovered":8003584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254244352,"data_stored":1254244352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119572,"ondisk_log_size":119572,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735112,"kb_used_data":324408,"kb_used_omap":4,"kb_used_meta":410683,"kb_avail":93636728,"statfs":{"total":96636764160,"available":95884009472,"internally_reserved":0,"allocated":332193792,"data_stored":332028803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":420540112},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740584,"kb_used_data":322072,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93631256,"statfs":{"total":96636764160,"available":95878406144,"internally_reserved":0,"allocated":329801728,"data_stored":329636739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541320,"kb_used_data":257016,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93830520,"statfs":{"total":96636764160,"available":96082452480,"internally_reserved":0,"allocated":263184384,"data_stored":263019395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743740,"kb_used_data":324780,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93628100,"statfs":{"total":96636764160,"available":95875174400,"internally_reserved":0,"allocated":332574720,"data_stored":332409731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":428994254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,5,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":331501568,"data_stored":331501568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262877184,"data_stored":262877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328908800,"data_stored":328908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":330956800,"data_stored":330956800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:45.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:46.441 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:46.442 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:46.460 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T21:08:46.037078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653899566,"num_objects":79827,"num_object_clones":0,"num_object_copies":319308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114585,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120498,"ondisk_log_size":120498,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741752,"kb_used_data":1217592,"kb_used_omap":14,"kb_used_meta":1524081,"kb_avail":374745608,"statfs":{"total":386547056640,"available":383739502592,"internally_reserved":0,"allocated":1246814208,"data_stored":1246154252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1560659506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,1,8,18,9,19,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27951104,"num_objects":-3412,"num_object_clones":0,"num_object_copies":-13648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004416"},"pg_stats":[{"pgid":"1.f","version":"33'7400","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.569405+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:45.569405+0000","last_peered":"2024-09-15T21:08:45.569405+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:45.569405+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7400,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2676}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7595","reported_seq":9704,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.051969+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:45.051969+0000","last_peered":"2024-09-15T21:08:45.051969+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:45.051969+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7427","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.672893+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:42.672893+0000","last_peered":"2024-09-15T21:08:42.672893+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:42.672893+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40999582,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7426,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7490","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.975236+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:44.975236+0000","last_peered":"2024-09-15T21:08:44.975236+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:44.975236+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40678668,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7611","reported_seq":7667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.623628+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:45.623628+0000","last_peered":"2024-09-15T21:08:45.623628+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:45.623628+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7611,"log_dups_size":0,"ondisk_log_size":7611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7505","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.562426+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:45.562426+0000","last_peered":"2024-09-15T21:08:45.562426+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:45.562426+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7722","reported_seq":7792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.947125+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:44.947125+0000","last_peered":"2024-09-15T21:08:44.947125+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:44.947125+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366796,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7710","reported_seq":7779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.981037+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:44.981037+0000","last_peered":"2024-09-15T21:08:44.981037+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:44.981037+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41374988,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7554","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.666269+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:45.666269+0000","last_peered":"2024-09-15T21:08:45.666269+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:45.666269+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40118808,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7499","reported_seq":7580,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.686389+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:42.686389+0000","last_peered":"2024-09-15T21:08:42.686389+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:42.686389+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473292,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2728}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7460","reported_seq":12123,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:45.550668+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:45.550668+0000","last_peered":"2024-09-15T21:08:45.550668+0000","last_clean":"2024-09-15T21:08:45.550668+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:45.550668+0000","last_undegraded":"2024-09-15T21:08:45.550668+0000","last_fullsized":"2024-09-15T21:08:45.550668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40140800,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7469","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.593468+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:45.593468+0000","last_peered":"2024-09-15T21:08:45.593468+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:45.593468+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40614534,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7455","reported_seq":7535,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.523864+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:42.523864+0000","last_peered":"2024-09-15T21:08:42.523864+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:42.523864+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41335430,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7454,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7476","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.511717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:45.511717+0000","last_peered":"2024-09-15T21:08:45.511717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:45.511717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40596748,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7567","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.747508+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:42.747508+0000","last_peered":"2024-09-15T21:08:42.747508+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:42.747508+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41891084,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7558","reported_seq":7625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.992320+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:44.992320+0000","last_peered":"2024-09-15T21:08:44.992320+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:44.992320+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40883468,"num_objects":4991,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7556,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653899566,"num_objects":79827,"num_object_clones":0,"num_object_copies":319308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114585,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1244209152,"data_stored":1244209152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120498,"ondisk_log_size":120498,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733452,"kb_used_data":318588,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93638388,"statfs":{"total":96636764160,"available":95885709312,"internally_reserved":0,"allocated":326234112,"data_stored":326069123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,9,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740584,"kb_used_data":322072,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93631256,"statfs":{"total":96636764160,"available":95878406144,"internally_reserved":0,"allocated":329801728,"data_stored":329636739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541320,"kb_used_data":257016,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93830520,"statfs":{"total":96636764160,"available":96082452480,"internally_reserved":0,"allocated":263184384,"data_stored":263019395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726396,"kb_used_data":319916,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93645444,"statfs":{"total":96636764160,"available":95892934656,"internally_reserved":0,"allocated":327593984,"data_stored":327428995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":416214734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,6,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326475776,"data_stored":326475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262877184,"data_stored":262877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328908800,"data_stored":328908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325947392,"data_stored":325947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:46.461 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7400", 'reported_seq': 7446, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.569405+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:45.569405+0000', 'last_peered': '2024-09-15T21:08:45.569405+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:45.569405+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7400, 'log_dups_size': 0, 'ondisk_log_size': 7400, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40566784, 'num_objects': 4952, 'num_object_clones': 0, 'num_object_copies': 19808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4952, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7400, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2676}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7595", 'reported_seq': 9704, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.051969+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:45.051969+0000', 'last_peered': '2024-09-15T21:08:45.051969+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:45.051969+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7595, 'log_dups_size': 0, 'ondisk_log_size': 7595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40604940, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1362, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7593, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1022, 'num_bytes_recovered': 8339456, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3595}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7427", 'reported_seq': 7507, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:42.672893+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:42.672893+0000', 'last_peered': '2024-09-15T21:08:42.672893+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:42.672893+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7427, 'log_dups_size': 0, 'ondisk_log_size': 7427, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40999582, 'num_objects': 5006, 'num_object_clones': 0, 'num_object_copies': 20024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5006, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7426, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2748}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7490", 'reported_seq': 7559, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:44.975236+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:44.975236+0000', 'last_peered': '2024-09-15T21:08:44.975236+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:44.975236+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7490, 'log_dups_size': 0, 'ondisk_log_size': 7490, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40678668, 'num_objects': 4966, 'num_object_clones': 0, 'num_object_copies': 19864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4966, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7488, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:46.462 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7611", 'reported_seq': 7667, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.623628+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:45.623628+0000', 'last_peered': '2024-09-15T21:08:45.623628+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:45.623628+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7611, 'log_dups_size': 0, 'ondisk_log_size': 7611, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41350412, 'num_objects': 5048, 'num_object_clones': 0, 'num_object_copies': 20192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5048, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7609, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2737}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7505", 'reported_seq': 7550, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.562426+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:45.562426+0000', 'last_peered': '2024-09-15T21:08:45.562426+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:45.562426+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7505, 'log_dups_size': 0, 'ondisk_log_size': 7505, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39903232, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7505, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7722", 'reported_seq': 7792, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:44.947125+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:44.947125+0000', 'last_peered': '2024-09-15T21:08:44.947125+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:44.947125+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7722, 'log_dups_size': 0, 'ondisk_log_size': 7722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41366796, 'num_objects': 5050, 'num_object_clones': 0, 'num_object_copies': 20200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7720, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2684}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7710", 'reported_seq': 7779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:44.981037+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:44.981037+0000', 'last_peered': '2024-09-15T21:08:44.981037+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:44.981037+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7710, 'log_dups_size': 0, 'ondisk_log_size': 7710, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41374988, 'num_objects': 5051, 'num_object_clones': 0, 'num_object_copies': 20204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7708, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:46.463 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7554", 'reported_seq': 7614, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.666269+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:45.666269+0000', 'last_peered': '2024-09-15T21:08:45.666269+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:45.666269+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7554, 'log_dups_size': 0, 'ondisk_log_size': 7554, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40118808, 'num_objects': 4898, 'num_object_clones': 0, 'num_object_copies': 19592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7551, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2566}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7499", 'reported_seq': 7580, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:42.686389+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:42.686389+0000', 'last_peered': '2024-09-15T21:08:42.686389+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:42.686389+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7499, 'log_dups_size': 0, 'ondisk_log_size': 7499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41473292, 'num_objects': 5063, 'num_object_clones': 0, 'num_object_copies': 20252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5063, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7497, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2728}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7469", 'reported_seq': 7509, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.593468+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:45.593468+0000', 'last_peered': '2024-09-15T21:08:45.593468+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:45.593468+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7469, 'log_dups_size': 0, 'ondisk_log_size': 7469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40614534, 'num_objects': 4958, 'num_object_clones': 0, 'num_object_copies': 19832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4958, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7468, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2702}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7455", 'reported_seq': 7535, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:42.523864+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:42.523864+0000', 'last_peered': '2024-09-15T21:08:42.523864+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:42.523864+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7455, 'log_dups_size': 0, 'ondisk_log_size': 7455, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41335430, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 20184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7454, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2760}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7476", 'reported_seq': 7533, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:45.511717+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:45.511717+0000', 'last_peered': '2024-09-15T21:08:45.511717+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:45.511717+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7476, 'log_dups_size': 0, 'ondisk_log_size': 7476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40596748, 'num_objects': 4956, 'num_object_clones': 0, 'num_object_copies': 19824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7474, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:46.464 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:46.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7567", 'reported_seq': 7650, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:42.747508+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:42.747508+0000', 'last_peered': '2024-09-15T21:08:42.747508+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:42.747508+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7567, 'log_dups_size': 0, 'ondisk_log_size': 7567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41891084, 'num_objects': 5114, 'num_object_clones': 0, 'num_object_copies': 20456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5114, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7565, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:46.465 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:46.465 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7558", 'reported_seq': 7625, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:44.992320+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:44.992320+0000', 'last_peered': '2024-09-15T21:08:44.992320+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:44.992320+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7558, 'log_dups_size': 0, 'ondisk_log_size': 7558, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40883468, 'num_objects': 4991, 'num_object_clones': 0, 'num_object_copies': 19964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4991, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7556, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:46.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:47.178 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:47.178 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33803/319308 objects degraded (10.586%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":392,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":79827,"data_bytes":653899566,"bytes_used":2807554048,"bytes_avail":383739502592,"bytes_total":386547056640,"degraded_objects":33803,"degraded_total":319308,"degraded_ratio":0.10586330439575581,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":89396,"recovering_keys_per_sec":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":284},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:47.201 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:47.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:47.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:47.872 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:47.901 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-15T21:08:46.037078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653899566,"num_objects":79827,"num_object_clones":0,"num_object_copies":319308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114585,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120498,"ondisk_log_size":120498,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741752,"kb_used_data":1217592,"kb_used_omap":14,"kb_used_meta":1524081,"kb_avail":374745608,"statfs":{"total":386547056640,"available":383739502592,"internally_reserved":0,"allocated":1246814208,"data_stored":1246154252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1560659506},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,1,8,18,9,19,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27951104,"num_objects":-3412,"num_object_clones":0,"num_object_copies":-13648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":1073152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004416"},"pg_stats":[{"pgid":"1.f","version":"33'7400","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.569405+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:45.569405+0000","last_peered":"2024-09-15T21:08:45.569405+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:45.569405+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7400,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2676}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7595","reported_seq":9704,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.051969+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:45.051969+0000","last_peered":"2024-09-15T21:08:45.051969+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:45.051969+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7427","reported_seq":7507,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.672893+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:42.672893+0000","last_peered":"2024-09-15T21:08:42.672893+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:42.672893+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7427,"log_dups_size":0,"ondisk_log_size":7427,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40999582,"num_objects":5006,"num_object_clones":0,"num_object_copies":20024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5006,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7426,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7490","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.975236+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:44.975236+0000","last_peered":"2024-09-15T21:08:44.975236+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:44.975236+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40678668,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7611","reported_seq":7667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.623628+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:45.623628+0000","last_peered":"2024-09-15T21:08:45.623628+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:45.623628+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7611,"log_dups_size":0,"ondisk_log_size":7611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7505","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.562426+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:45.562426+0000","last_peered":"2024-09-15T21:08:45.562426+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:45.562426+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7722","reported_seq":7792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.947125+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:44.947125+0000","last_peered":"2024-09-15T21:08:44.947125+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:44.947125+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366796,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7710","reported_seq":7779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.981037+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:44.981037+0000","last_peered":"2024-09-15T21:08:44.981037+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:44.981037+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41374988,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7554","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.666269+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:45.666269+0000","last_peered":"2024-09-15T21:08:45.666269+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:45.666269+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40118808,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7499","reported_seq":7580,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.686389+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:42.686389+0000","last_peered":"2024-09-15T21:08:42.686389+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:42.686389+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7499,"log_dups_size":0,"ondisk_log_size":7499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41473292,"num_objects":5063,"num_object_clones":0,"num_object_copies":20252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2728}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7460","reported_seq":12123,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:45.550668+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:45.550668+0000","last_peered":"2024-09-15T21:08:45.550668+0000","last_clean":"2024-09-15T21:08:45.550668+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:45.550668+0000","last_undegraded":"2024-09-15T21:08:45.550668+0000","last_fullsized":"2024-09-15T21:08:45.550668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40140800,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7469","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.593468+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:45.593468+0000","last_peered":"2024-09-15T21:08:45.593468+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:45.593468+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40614534,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7455","reported_seq":7535,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.523864+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:42.523864+0000","last_peered":"2024-09-15T21:08:42.523864+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:42.523864+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7455,"log_dups_size":0,"ondisk_log_size":7455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41335430,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7454,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2760}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7476","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.511717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:45.511717+0000","last_peered":"2024-09-15T21:08:45.511717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:45.511717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40596748,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7567","reported_seq":7650,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:42.747508+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:42.747508+0000","last_peered":"2024-09-15T21:08:42.747508+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:42.747508+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41891084,"num_objects":5114,"num_object_clones":0,"num_object_copies":20456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7558","reported_seq":7625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.992320+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:44.992320+0000","last_peered":"2024-09-15T21:08:44.992320+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:44.992320+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40883468,"num_objects":4991,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7556,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":653899566,"num_objects":79827,"num_object_clones":0,"num_object_copies":319308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114585,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1244209152,"data_stored":1244209152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120498,"ondisk_log_size":120498,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733452,"kb_used_data":318588,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93638388,"statfs":{"total":96636764160,"available":95885709312,"internally_reserved":0,"allocated":326234112,"data_stored":326069123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,9,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740584,"kb_used_data":322072,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93631256,"statfs":{"total":96636764160,"available":95878406144,"internally_reserved":0,"allocated":329801728,"data_stored":329636739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,1,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051642,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":541320,"kb_used_data":257016,"kb_used_omap":4,"kb_used_meta":284283,"kb_avail":93830520,"statfs":{"total":96636764160,"available":96082452480,"internally_reserved":0,"allocated":263184384,"data_stored":263019395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":291106511},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726396,"kb_used_data":319916,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93645444,"statfs":{"total":96636764160,"available":95892934656,"internally_reserved":0,"allocated":327593984,"data_stored":327428995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":416214734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,6,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326475776,"data_stored":326475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262877184,"data_stored":262877184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":328908800,"data_stored":328908800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325947392,"data_stored":325947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:50.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:51.521 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:51.522 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:51.551 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-09-15T21:08:50.038658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651319086,"num_objects":79512,"num_object_clones":0,"num_object_copies":318048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114900,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120813,"ondisk_log_size":120813,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2762908,"kb_used_data":1212376,"kb_used_omap":14,"kb_used_meta":1550449,"kb_avail":374724452,"statfs":{"total":386547056640,"available":383717838848,"internally_reserved":0,"allocated":1241473024,"data_stored":1240813068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1587660338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,11,18,15,21,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20340736,"num_objects":-2483,"num_object_clones":0,"num_object_copies":-9932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":729088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004200"},"pg_stats":[{"pgid":"1.f","version":"33'7400","reported_seq":7446,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.569405+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:45.569405+0000","last_peered":"2024-09-15T21:08:45.569405+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:45.569405+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7400,"log_dups_size":0,"ondisk_log_size":7400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40566784,"num_objects":4952,"num_object_clones":0,"num_object_copies":19808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7400,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2676}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7595","reported_seq":9704,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.051969+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:45.051969+0000","last_peered":"2024-09-15T21:08:45.051969+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:45.051969+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7595,"log_dups_size":0,"ondisk_log_size":7595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40604940,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1362,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7593,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3595}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7505","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.828363+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:47.828363+0000","last_peered":"2024-09-15T21:08:47.828363+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:47.828363+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40360606,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7504,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7490","reported_seq":7559,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.975236+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:44.975236+0000","last_peered":"2024-09-15T21:08:44.975236+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:44.975236+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7490,"log_dups_size":0,"ondisk_log_size":7490,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40678668,"num_objects":4966,"num_object_clones":0,"num_object_copies":19864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7488,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7611","reported_seq":7667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.623628+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:45.623628+0000","last_peered":"2024-09-15T21:08:45.623628+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:45.623628+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7611,"log_dups_size":0,"ondisk_log_size":7611,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41350412,"num_objects":5048,"num_object_clones":0,"num_object_copies":20192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7609,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2737}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7505","reported_seq":7550,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.562426+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:45.562426+0000","last_peered":"2024-09-15T21:08:45.562426+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:45.562426+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39903232,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7505,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7722","reported_seq":7792,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.947125+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:44.947125+0000","last_peered":"2024-09-15T21:08:44.947125+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:44.947125+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7722,"log_dups_size":0,"ondisk_log_size":7722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41366796,"num_objects":5050,"num_object_clones":0,"num_object_copies":20200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7720,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2684}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7710","reported_seq":7779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.981037+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:44.981037+0000","last_peered":"2024-09-15T21:08:44.981037+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:44.981037+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7710,"log_dups_size":0,"ondisk_log_size":7710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41374988,"num_objects":5051,"num_object_clones":0,"num_object_copies":20204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7708,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2577}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7554","reported_seq":7614,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.666269+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:45.666269+0000","last_peered":"2024-09-15T21:08:45.666269+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:45.666269+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40118808,"num_objects":4898,"num_object_clones":0,"num_object_copies":19592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7551,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2566}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7579","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.780409+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:47.780409+0000","last_peered":"2024-09-15T21:08:47.780409+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:47.780409+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40817932,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7460","reported_seq":12123,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:45.550668+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:45.550668+0000","last_peered":"2024-09-15T21:08:45.550668+0000","last_clean":"2024-09-15T21:08:45.550668+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:45.550668+0000","last_undegraded":"2024-09-15T21:08:45.550668+0000","last_fullsized":"2024-09-15T21:08:45.550668+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7460,"log_dups_size":0,"ondisk_log_size":7460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40140800,"num_objects":4900,"num_object_clones":0,"num_object_copies":19600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7469","reported_seq":7509,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.593468+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:45.593468+0000","last_peered":"2024-09-15T21:08:45.593468+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:45.593468+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40614534,"num_objects":4958,"num_object_clones":0,"num_object_copies":19832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2702}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7544","reported_seq":7624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.799108+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:47.799108+0000","last_peered":"2024-09-15T21:08:47.799108+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:47.799108+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606342,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7476","reported_seq":7533,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:45.511717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:45.511717+0000","last_peered":"2024-09-15T21:08:45.511717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:45.511717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7476,"log_dups_size":0,"ondisk_log_size":7476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40596748,"num_objects":4956,"num_object_clones":0,"num_object_copies":19824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7635","reported_seq":7718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.699535+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:47.699535+0000","last_peered":"2024-09-15T21:08:47.699535+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:47.699535+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41334028,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7558","reported_seq":7625,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:44.992320+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:44.992320+0000","last_peered":"2024-09-15T21:08:44.992320+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:44.992320+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7558,"log_dups_size":0,"ondisk_log_size":7558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40883468,"num_objects":4991,"num_object_clones":0,"num_object_copies":19964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7556,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":651319086,"num_objects":79512,"num_object_clones":0,"num_object_copies":318048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33803,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79512,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":114900,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":8339456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1238990848,"data_stored":1238990848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120813,"ondisk_log_size":120813,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733452,"kb_used_data":318588,"kb_used_omap":4,"kb_used_meta":414843,"kb_avail":93638388,"statfs":{"total":96636764160,"available":95885709312,"internally_reserved":0,"allocated":326234112,"data_stored":326069123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":424799952},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,9,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757792,"kb_used_data":317068,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93614048,"statfs":{"total":96636764160,"available":95860785152,"internally_reserved":0,"allocated":324677632,"data_stored":324512643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545268,"kb_used_data":256804,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93826572,"statfs":{"total":96636764160,"available":96078409728,"internally_reserved":0,"allocated":262967296,"data_stored":262802307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247880,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726396,"kb_used_data":319916,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93645444,"statfs":{"total":96636764160,"available":95892934656,"internally_reserved":0,"allocated":327593984,"data_stored":327428995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":416214734},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,6,3,7,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":326475776,"data_stored":326475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262672384,"data_stored":262672384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323895296,"data_stored":323895296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":325947392,"data_stored":325947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:51.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:52.207 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:52.207 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:52.232 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-15T21:08:52.039427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643528494,"num_objects":78561,"num_object_clones":0,"num_object_copies":314244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115851,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121764,"ondisk_log_size":121764,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798512,"kb_used_data":1203556,"kb_used_omap":14,"kb_used_meta":1594865,"kb_avail":374688848,"statfs":{"total":386547056640,"available":383681380352,"internally_reserved":0,"allocated":1232441344,"data_stored":1231781388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1633142322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,12,9,24,18,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28131328,"num_objects":-3434,"num_object_clones":0,"num_object_copies":-13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004251"},"pg_stats":[{"pgid":"1.f","version":"33'7478","reported_seq":7524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.615865+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:50.615865+0000","last_peered":"2024-09-15T21:08:50.615865+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:50.615865+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39927808,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7666","reported_seq":9873,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.124066+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:50.124066+0000","last_peered":"2024-09-15T21:08:50.124066+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:50.124066+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023308,"num_objects":4886,"num_object_clones":0,"num_object_copies":19544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7505","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.828363+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:47.828363+0000","last_peered":"2024-09-15T21:08:47.828363+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:47.828363+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40360606,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7504,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7567","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.096920+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:50.096920+0000","last_peered":"2024-09-15T21:08:50.096920+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:50.096920+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047884,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2555}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7688","reported_seq":7744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.585624+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:50.585624+0000","last_peered":"2024-09-15T21:08:50.585624+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:50.585624+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7583","reported_seq":7628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.691218+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:50.691218+0000","last_peered":"2024-09-15T21:08:50.691218+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:50.691218+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7804","reported_seq":7874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.086537+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:50.086537+0000","last_peered":"2024-09-15T21:08:50.086537+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:50.086537+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7804,"log_dups_size":0,"ondisk_log_size":7804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40695052,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7802,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7797","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:49.842351+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:49.842351+0000","last_peered":"2024-09-15T21:08:49.842351+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:49.842351+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7797,"log_dups_size":0,"ondisk_log_size":7797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662284,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7795,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7634","reported_seq":7694,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.583190+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:50.583190+0000","last_peered":"2024-09-15T21:08:50.583190+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:50.583190+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39463448,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7579","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.780409+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:47.780409+0000","last_peered":"2024-09-15T21:08:47.780409+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:47.780409+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40817932,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7536","reported_seq":12198,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:50.719006+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:50.719006+0000","last_peered":"2024-09-15T21:08:50.719006+0000","last_clean":"2024-09-15T21:08:50.719006+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:50.719006+0000","last_undegraded":"2024-09-15T21:08:50.719006+0000","last_fullsized":"2024-09-15T21:08:50.719006+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":4824,"num_object_clones":0,"num_object_copies":19296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1647,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7548","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.606966+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:50.606966+0000","last_peered":"2024-09-15T21:08:50.606966+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:50.606966+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39967366,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7544","reported_seq":7624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.799108+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:47.799108+0000","last_peered":"2024-09-15T21:08:47.799108+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:47.799108+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606342,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7567","reported_seq":7624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.663381+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:50.663381+0000","last_peered":"2024-09-15T21:08:50.663381+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:50.663381+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39851276,"num_objects":4865,"num_object_clones":0,"num_object_copies":19460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7635","reported_seq":7718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.699535+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:47.699535+0000","last_peered":"2024-09-15T21:08:47.699535+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:47.699535+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41334028,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7633","reported_seq":7702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.115019+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:50.115019+0000","last_peered":"2024-09-15T21:08:50.115019+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:50.115019+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269068,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":643528494,"num_objects":78561,"num_object_clones":0,"num_object_copies":314244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115851,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1228939264,"data_stored":1228939264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121764,"ondisk_log_size":121764,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751696,"kb_used_data":314624,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93620144,"statfs":{"total":96636764160,"available":95867027456,"internally_reserved":0,"allocated":322174976,"data_stored":322009987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,7,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757792,"kb_used_data":317068,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93614048,"statfs":{"total":96636764160,"available":95860785152,"internally_reserved":0,"allocated":324677632,"data_stored":324512643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545268,"kb_used_data":256804,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93826572,"statfs":{"total":96636764160,"available":96078409728,"internally_reserved":0,"allocated":262967296,"data_stored":262802307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743756,"kb_used_data":315060,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93628084,"statfs":{"total":96636764160,"available":95875158016,"internally_reserved":0,"allocated":322621440,"data_stored":322456451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,11,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321515520,"data_stored":321515520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262672384,"data_stored":262672384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323895296,"data_stored":323895296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320856064,"data_stored":320856064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7478", 'reported_seq': 7524, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.615865+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:50.615865+0000', 'last_peered': '2024-09-15T21:08:50.615865+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:50.615865+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7478, 'log_dups_size': 0, 'ondisk_log_size': 7478, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39927808, 'num_objects': 4874, 'num_object_clones': 0, 'num_object_copies': 19496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4874, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7478, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2598}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7666", 'reported_seq': 9873, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.124066+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:50.124066+0000', 'last_peered': '2024-09-15T21:08:50.124066+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:50.124066+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7666, 'log_dups_size': 0, 'ondisk_log_size': 7666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40023308, 'num_objects': 4886, 'num_object_clones': 0, 'num_object_copies': 19544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7664, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1071, 'num_bytes_recovered': 8740864, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3573}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7505", 'reported_seq': 7585, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:47.828363+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:47.828363+0000', 'last_peered': '2024-09-15T21:08:47.828363+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:47.828363+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7505, 'log_dups_size': 0, 'ondisk_log_size': 7505, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40360606, 'num_objects': 4928, 'num_object_clones': 0, 'num_object_copies': 19712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4928, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7504, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2670}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7567", 'reported_seq': 7636, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.096920+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:50.096920+0000', 'last_peered': '2024-09-15T21:08:50.096920+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:50.096920+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7567, 'log_dups_size': 0, 'ondisk_log_size': 7567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40047884, 'num_objects': 4889, 'num_object_clones': 0, 'num_object_copies': 19556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7565, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2555}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:52.234 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7688", 'reported_seq': 7744, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.585624+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:50.585624+0000', 'last_peered': '2024-09-15T21:08:50.585624+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:50.585624+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7688, 'log_dups_size': 0, 'ondisk_log_size': 7688, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40719628, 'num_objects': 4971, 'num_object_clones': 0, 'num_object_copies': 19884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7686, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2660}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7583", 'reported_seq': 7628, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.691218+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:50.691218+0000', 'last_peered': '2024-09-15T21:08:50.691218+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:50.691218+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7583, 'log_dups_size': 0, 'ondisk_log_size': 7583, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39264256, 'num_objects': 4793, 'num_object_clones': 0, 'num_object_copies': 19172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7583, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7804", 'reported_seq': 7874, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.086537+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:50.086537+0000', 'last_peered': '2024-09-15T21:08:50.086537+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:50.086537+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7804, 'log_dups_size': 0, 'ondisk_log_size': 7804, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40695052, 'num_objects': 4968, 'num_object_clones': 0, 'num_object_copies': 19872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7802, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2602}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7797", 'reported_seq': 7866, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:49.842351+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:49.842351+0000', 'last_peered': '2024-09-15T21:08:49.842351+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:49.842351+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7797, 'log_dups_size': 0, 'ondisk_log_size': 7797, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40662284, 'num_objects': 4964, 'num_object_clones': 0, 'num_object_copies': 19856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7795, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2490}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7634", 'reported_seq': 7694, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.583190+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:50.583190+0000', 'last_peered': '2024-09-15T21:08:50.583190+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:50.583190+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7634, 'log_dups_size': 0, 'ondisk_log_size': 7634, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39463448, 'num_objects': 4818, 'num_object_clones': 0, 'num_object_copies': 19272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7631, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2486}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.235 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7579", 'reported_seq': 7659, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:47.780409+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:47.780409+0000', 'last_peered': '2024-09-15T21:08:47.780409+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:47.780409+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7579, 'log_dups_size': 0, 'ondisk_log_size': 7579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40817932, 'num_objects': 4983, 'num_object_clones': 0, 'num_object_copies': 19932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4983, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7577, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2648}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7548", 'reported_seq': 7588, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.606966+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:50.606966+0000', 'last_peered': '2024-09-15T21:08:50.606966+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:50.606966+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7548, 'log_dups_size': 0, 'ondisk_log_size': 7548, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39967366, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7547, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7544", 'reported_seq': 7624, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:47.799108+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:47.799108+0000', 'last_peered': '2024-09-15T21:08:47.799108+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:47.799108+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7544, 'log_dups_size': 0, 'ondisk_log_size': 7544, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40606342, 'num_objects': 4957, 'num_object_clones': 0, 'num_object_copies': 19828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7543, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7567", 'reported_seq': 7624, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.663381+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:50.663381+0000', 'last_peered': '2024-09-15T21:08:50.663381+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:50.663381+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7567, 'log_dups_size': 0, 'ondisk_log_size': 7567, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39851276, 'num_objects': 4865, 'num_object_clones': 0, 'num_object_copies': 19460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7565, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7635", 'reported_seq': 7718, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:47.699535+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:47.699535+0000', 'last_peered': '2024-09-15T21:08:47.699535+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:47.699535+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7635, 'log_dups_size': 0, 'ondisk_log_size': 7635, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 41334028, 'num_objects': 5046, 'num_object_clones': 0, 'num_object_copies': 20184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5046, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7633, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:52.236 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:52.237 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7633", 'reported_seq': 7702, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:50.115019+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:50.115019+0000', 'last_peered': '2024-09-15T21:08:50.115019+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:50.115019+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7633, 'log_dups_size': 0, 'ondisk_log_size': 7633, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40269068, 'num_objects': 4916, 'num_object_clones': 0, 'num_object_copies': 19664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7631, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:52.237 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:52.995 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:52.995 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33754/314244 objects degraded (10.741%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":398,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":78561,"data_bytes":643528494,"bytes_used":2865676288,"bytes_avail":383681380352,"bytes_total":386547056640,"degraded_objects":33754,"degraded_total":314244,"degraded_ratio":0.107413347589771,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":94174,"recovering_keys_per_sec":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":286},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:53.032 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:53.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:53.602 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:53.602 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:53.625 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-15T21:08:52.039427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":643528494,"num_objects":78561,"num_object_clones":0,"num_object_copies":314244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115851,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121764,"ondisk_log_size":121764,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798512,"kb_used_data":1203556,"kb_used_omap":14,"kb_used_meta":1594865,"kb_avail":374688848,"statfs":{"total":386547056640,"available":383681380352,"internally_reserved":0,"allocated":1232441344,"data_stored":1231781388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1633142322},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,12,9,24,18,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-28131328,"num_objects":-3434,"num_object_clones":0,"num_object_copies":-13736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-138,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3434,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":1130496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004251"},"pg_stats":[{"pgid":"1.f","version":"33'7478","reported_seq":7524,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.615865+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:50.615865+0000","last_peered":"2024-09-15T21:08:50.615865+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:50.615865+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7478,"log_dups_size":0,"ondisk_log_size":7478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39927808,"num_objects":4874,"num_object_clones":0,"num_object_copies":19496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7478,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2598}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7666","reported_seq":9873,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.124066+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:50.124066+0000","last_peered":"2024-09-15T21:08:50.124066+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:50.124066+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7666,"log_dups_size":0,"ondisk_log_size":7666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40023308,"num_objects":4886,"num_object_clones":0,"num_object_copies":19544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7664,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1071,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3573}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7505","reported_seq":7585,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.828363+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:47.828363+0000","last_peered":"2024-09-15T21:08:47.828363+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:47.828363+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40360606,"num_objects":4928,"num_object_clones":0,"num_object_copies":19712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7504,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2670}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7567","reported_seq":7636,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.096920+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:50.096920+0000","last_peered":"2024-09-15T21:08:50.096920+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:50.096920+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047884,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2555}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7688","reported_seq":7744,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.585624+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:50.585624+0000","last_peered":"2024-09-15T21:08:50.585624+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:50.585624+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7688,"log_dups_size":0,"ondisk_log_size":7688,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40719628,"num_objects":4971,"num_object_clones":0,"num_object_copies":19884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7686,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2660}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7583","reported_seq":7628,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.691218+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:50.691218+0000","last_peered":"2024-09-15T21:08:50.691218+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:50.691218+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39264256,"num_objects":4793,"num_object_clones":0,"num_object_copies":19172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7804","reported_seq":7874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.086537+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:50.086537+0000","last_peered":"2024-09-15T21:08:50.086537+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:50.086537+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7804,"log_dups_size":0,"ondisk_log_size":7804,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40695052,"num_objects":4968,"num_object_clones":0,"num_object_copies":19872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7802,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2602}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7797","reported_seq":7866,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:49.842351+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:49.842351+0000","last_peered":"2024-09-15T21:08:49.842351+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:49.842351+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7797,"log_dups_size":0,"ondisk_log_size":7797,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40662284,"num_objects":4964,"num_object_clones":0,"num_object_copies":19856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7795,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2490}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7634","reported_seq":7694,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.583190+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:50.583190+0000","last_peered":"2024-09-15T21:08:50.583190+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:50.583190+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7634,"log_dups_size":0,"ondisk_log_size":7634,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39463448,"num_objects":4818,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2486}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7579","reported_seq":7659,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.780409+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:47.780409+0000","last_peered":"2024-09-15T21:08:47.780409+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:47.780409+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7579,"log_dups_size":0,"ondisk_log_size":7579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40817932,"num_objects":4983,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7577,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2648}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7536","reported_seq":12198,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:50.719006+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:50.719006+0000","last_peered":"2024-09-15T21:08:50.719006+0000","last_clean":"2024-09-15T21:08:50.719006+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:50.719006+0000","last_undegraded":"2024-09-15T21:08:50.719006+0000","last_fullsized":"2024-09-15T21:08:50.719006+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39518208,"num_objects":4824,"num_object_clones":0,"num_object_copies":19296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1647,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7548","reported_seq":7588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.606966+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:50.606966+0000","last_peered":"2024-09-15T21:08:50.606966+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:50.606966+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7548,"log_dups_size":0,"ondisk_log_size":7548,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39967366,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7547,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2623}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7544","reported_seq":7624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.799108+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:47.799108+0000","last_peered":"2024-09-15T21:08:47.799108+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:47.799108+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7544,"log_dups_size":0,"ondisk_log_size":7544,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40606342,"num_objects":4957,"num_object_clones":0,"num_object_copies":19828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7543,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2671}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7567","reported_seq":7624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.663381+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:50.663381+0000","last_peered":"2024-09-15T21:08:50.663381+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:50.663381+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7567,"log_dups_size":0,"ondisk_log_size":7567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39851276,"num_objects":4865,"num_object_clones":0,"num_object_copies":19460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7565,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7635","reported_seq":7718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:47.699535+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:47.699535+0000","last_peered":"2024-09-15T21:08:47.699535+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:47.699535+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7635,"log_dups_size":0,"ondisk_log_size":7635,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41334028,"num_objects":5046,"num_object_clones":0,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7633,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2765}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7633","reported_seq":7702,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:50.115019+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:50.115019+0000","last_peered":"2024-09-15T21:08:50.115019+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:50.115019+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7633,"log_dups_size":0,"ondisk_log_size":7633,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40269068,"num_objects":4916,"num_object_clones":0,"num_object_copies":19664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7631,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2601}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":643528494,"num_objects":78561,"num_object_clones":0,"num_object_copies":314244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115851,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1093,"num_bytes_recovered":8740864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1228939264,"data_stored":1228939264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121764,"ondisk_log_size":121764,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751696,"kb_used_data":314624,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93620144,"statfs":{"total":96636764160,"available":95867027456,"internally_reserved":0,"allocated":322174976,"data_stored":322009987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,7,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757792,"kb_used_data":317068,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":93614048,"statfs":{"total":96636764160,"available":95860785152,"internally_reserved":0,"allocated":324677632,"data_stored":324512643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":451279301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,4,1,8,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051643,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":545268,"kb_used_data":256804,"kb_used_omap":4,"kb_used_meta":288443,"kb_avail":93826572,"statfs":{"total":96636764160,"available":96078409728,"internally_reserved":0,"allocated":262967296,"data_stored":262802307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":295366351},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743756,"kb_used_data":315060,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93628084,"statfs":{"total":96636764160,"available":95875158016,"internally_reserved":0,"allocated":322621440,"data_stored":322456451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":438955726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,11,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":321515520,"data_stored":321515520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262672384,"data_stored":262672384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":323895296,"data_stored":323895296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":320856064,"data_stored":320856064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:56.627 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:57.225 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:57.226 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:57.244 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T21:08:56.041032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633624366,"num_objects":77352,"num_object_clones":0,"num_object_copies":309408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117060,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122973,"ondisk_log_size":122973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766756,"kb_used_data":1188432,"kb_used_omap":14,"kb_used_meta":1578225,"kb_avail":374720604,"statfs":{"total":386547056640,"available":383713898496,"internally_reserved":0,"allocated":1216954368,"data_stored":1216294412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1616102962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,7,16,8,8,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27860992,"num_objects":-3401,"num_object_clones":0,"num_object_copies":-13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004470"},"pg_stats":[{"pgid":"1.f","version":"33'7569","reported_seq":7613,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.784019+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:55.784019+0000","last_peered":"2024-09-15T21:08:55.784019+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:55.784019+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39182336,"num_objects":4783,"num_object_clones":0,"num_object_copies":19132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7750","reported_seq":10042,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.068615+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:55.068615+0000","last_peered":"2024-09-15T21:08:55.068615+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:55.068615+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7562","reported_seq":7642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.524588+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:52.524588+0000","last_peered":"2024-09-15T21:08:52.524588+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:52.524588+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39893662,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7561,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7636","reported_seq":7705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:54.965440+0000","last_peered":"2024-09-15T21:08:54.965440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:54.965440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482636,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7780","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.732372+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:55.732372+0000","last_peered":"2024-09-15T21:08:55.732372+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:55.732372+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7653","reported_seq":7698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.458770+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:55.458770+0000","last_peered":"2024-09-15T21:08:55.458770+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:55.458770+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38690816,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7653,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7871","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965548+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:54.965548+0000","last_peered":"2024-09-15T21:08:54.965548+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:54.965548+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146188,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7872","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.086032+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:55.086032+0000","last_peered":"2024-09-15T21:08:55.086032+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:55.086032+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047884,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7703","reported_seq":7763,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.745404+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:55.745404+0000","last_peered":"2024-09-15T21:08:55.745404+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:55.745404+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38898200,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7658","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.800514+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:52.800514+0000","last_peered":"2024-09-15T21:08:52.800514+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:52.800514+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40170764,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7619","reported_seq":12281,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:55.756564+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:55.756564+0000","last_peered":"2024-09-15T21:08:55.756564+0000","last_clean":"2024-09-15T21:08:55.756564+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:55.756564+0000","last_undegraded":"2024-09-15T21:08:55.756564+0000","last_fullsized":"2024-09-15T21:08:55.756564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38838272,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1730,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7622","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674748+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:55.674748+0000","last_peered":"2024-09-15T21:08:55.674748+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:55.674748+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2549}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7629","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.866761+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:52.866761+0000","last_peered":"2024-09-15T21:08:52.866761+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:52.866761+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39910022,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7632","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:55.674131+0000","last_peered":"2024-09-15T21:08:55.674131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:55.674131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39318796,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7703","reported_seq":7786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.831157+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:52.831157+0000","last_peered":"2024-09-15T21:08:52.831157+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:52.831157+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40776972,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7714","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.120899+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:55.120899+0000","last_peered":"2024-09-15T21:08:55.120899+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:55.120899+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605516,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":633624366,"num_objects":77352,"num_object_clones":0,"num_object_copies":309408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117060,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1213566976,"data_stored":1213566976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122973,"ondisk_log_size":122973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734316,"kb_used_data":309704,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93637524,"statfs":{"total":96636764160,"available":95884824576,"internally_reserved":0,"allocated":317136896,"data_stored":316971907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,2,1,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740388,"kb_used_data":312148,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93631452,"statfs":{"total":96636764160,"available":95878606848,"internally_reserved":0,"allocated":319639552,"data_stored":319474563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549128,"kb_used_data":256504,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93822712,"statfs":{"total":96636764160,"available":96074457088,"internally_reserved":0,"allocated":262660096,"data_stored":262495107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742924,"kb_used_data":310076,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93628916,"statfs":{"total":96636764160,"available":95876009984,"internally_reserved":0,"allocated":317517824,"data_stored":317352835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,0,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316567552,"data_stored":316567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262361088,"data_stored":262361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318783488,"data_stored":318783488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315854848,"data_stored":315854848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:57.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:57.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:57.811 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-15T21:08:56.041032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633624366,"num_objects":77352,"num_object_clones":0,"num_object_copies":309408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117060,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122973,"ondisk_log_size":122973,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766756,"kb_used_data":1188432,"kb_used_omap":14,"kb_used_meta":1578225,"kb_avail":374720604,"statfs":{"total":386547056640,"available":383713898496,"internally_reserved":0,"allocated":1216954368,"data_stored":1216294412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1616102962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,7,16,8,8,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27860992,"num_objects":-3401,"num_object_clones":0,"num_object_copies":-13604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":1089536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004470"},"pg_stats":[{"pgid":"1.f","version":"33'7569","reported_seq":7613,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.784019+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:55.784019+0000","last_peered":"2024-09-15T21:08:55.784019+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:55.784019+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39182336,"num_objects":4783,"num_object_clones":0,"num_object_copies":19132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7750","reported_seq":10042,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.068615+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:55.068615+0000","last_peered":"2024-09-15T21:08:55.068615+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:55.068615+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7562","reported_seq":7642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.524588+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:52.524588+0000","last_peered":"2024-09-15T21:08:52.524588+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:52.524588+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7562,"log_dups_size":0,"ondisk_log_size":7562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39893662,"num_objects":4871,"num_object_clones":0,"num_object_copies":19484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4871,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7561,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2613}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7636","reported_seq":7705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:54.965440+0000","last_peered":"2024-09-15T21:08:54.965440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:54.965440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482636,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7780","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.732372+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:55.732372+0000","last_peered":"2024-09-15T21:08:55.732372+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:55.732372+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7653","reported_seq":7698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.458770+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:55.458770+0000","last_peered":"2024-09-15T21:08:55.458770+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:55.458770+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38690816,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7653,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7871","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965548+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:54.965548+0000","last_peered":"2024-09-15T21:08:54.965548+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:54.965548+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146188,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7872","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.086032+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:55.086032+0000","last_peered":"2024-09-15T21:08:55.086032+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:55.086032+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047884,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7703","reported_seq":7763,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.745404+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:55.745404+0000","last_peered":"2024-09-15T21:08:55.745404+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:55.745404+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38898200,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7658","reported_seq":7739,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.800514+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:52.800514+0000","last_peered":"2024-09-15T21:08:52.800514+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:52.800514+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7658,"log_dups_size":0,"ondisk_log_size":7658,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40170764,"num_objects":4904,"num_object_clones":0,"num_object_copies":19616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7656,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2569}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7619","reported_seq":12281,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:55.756564+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:55.756564+0000","last_peered":"2024-09-15T21:08:55.756564+0000","last_clean":"2024-09-15T21:08:55.756564+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:55.756564+0000","last_undegraded":"2024-09-15T21:08:55.756564+0000","last_fullsized":"2024-09-15T21:08:55.756564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38838272,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1730,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7622","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674748+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:55.674748+0000","last_peered":"2024-09-15T21:08:55.674748+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:55.674748+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2549}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7629","reported_seq":7708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.866761+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:52.866761+0000","last_peered":"2024-09-15T21:08:52.866761+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:52.866761+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7629,"log_dups_size":0,"ondisk_log_size":7629,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39910022,"num_objects":4872,"num_object_clones":0,"num_object_copies":19488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7628,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2586}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7632","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:55.674131+0000","last_peered":"2024-09-15T21:08:55.674131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:55.674131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39318796,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7703","reported_seq":7786,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:52.831157+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:52.831157+0000","last_peered":"2024-09-15T21:08:52.831157+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:52.831157+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40776972,"num_objects":4978,"num_object_clones":0,"num_object_copies":19912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2697}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7714","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.120899+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:55.120899+0000","last_peered":"2024-09-15T21:08:55.120899+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:55.120899+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605516,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":633624366,"num_objects":77352,"num_object_clones":0,"num_object_copies":309408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77352,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117060,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1213566976,"data_stored":1213566976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122973,"ondisk_log_size":122973,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734316,"kb_used_data":309704,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93637524,"statfs":{"total":96636764160,"available":95884824576,"internally_reserved":0,"allocated":317136896,"data_stored":316971907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,2,1,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740388,"kb_used_data":312148,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93631452,"statfs":{"total":96636764160,"available":95878606848,"internally_reserved":0,"allocated":319639552,"data_stored":319474563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,7,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549128,"kb_used_data":256504,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93822712,"statfs":{"total":96636764160,"available":96074457088,"internally_reserved":0,"allocated":262660096,"data_stored":262495107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742924,"kb_used_data":310076,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93628916,"statfs":{"total":96636764160,"available":95876009984,"internally_reserved":0,"allocated":317517824,"data_stored":317352835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,0,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316567552,"data_stored":316567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262361088,"data_stored":262361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":318783488,"data_stored":318783488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315854848,"data_stored":315854848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7569", 'reported_seq': 7613, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.784019+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:08:55.784019+0000', 'last_peered': '2024-09-15T21:08:55.784019+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:08:55.784019+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7569, 'log_dups_size': 0, 'ondisk_log_size': 7569, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39182336, 'num_objects': 4783, 'num_object_clones': 0, 'num_object_copies': 19132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4783, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7569, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2507}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7750", 'reported_seq': 10042, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.068615+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:08:55.068615+0000', 'last_peered': '2024-09-15T21:08:55.068615+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:08:55.068615+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7750, 'log_dups_size': 0, 'ondisk_log_size': 7750, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39335180, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1270, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7748, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1114, 'num_bytes_recovered': 9093120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3532}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7562", 'reported_seq': 7642, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:52.524588+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:52.524588+0000', 'last_peered': '2024-09-15T21:08:52.524588+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:52.524588+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7562, 'log_dups_size': 0, 'ondisk_log_size': 7562, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39893662, 'num_objects': 4871, 'num_object_clones': 0, 'num_object_copies': 19484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4871, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7561, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2613}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:57.840 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7636", 'reported_seq': 7705, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:54.965440+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:08:54.965440+0000', 'last_peered': '2024-09-15T21:08:54.965440+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:08:54.965440+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7636, 'log_dups_size': 0, 'ondisk_log_size': 7636, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39482636, 'num_objects': 4820, 'num_object_clones': 0, 'num_object_copies': 19280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7634, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7780", 'reported_seq': 7835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.732372+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:08:55.732372+0000', 'last_peered': '2024-09-15T21:08:55.732372+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:08:55.732372+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7780, 'log_dups_size': 0, 'ondisk_log_size': 7780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39965964, 'num_objects': 4879, 'num_object_clones': 0, 'num_object_copies': 19516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7778, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2568}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7653", 'reported_seq': 7698, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.458770+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:08:55.458770+0000', 'last_peered': '2024-09-15T21:08:55.458770+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:08:55.458770+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7653, 'log_dups_size': 0, 'ondisk_log_size': 7653, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38690816, 'num_objects': 4723, 'num_object_clones': 0, 'num_object_copies': 18892, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4723, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7653, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2407}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7871", 'reported_seq': 7941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:54.965548+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:08:54.965548+0000', 'last_peered': '2024-09-15T21:08:54.965548+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:08:54.965548+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7871, 'log_dups_size': 0, 'ondisk_log_size': 7871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40146188, 'num_objects': 4901, 'num_object_clones': 0, 'num_object_copies': 19604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7869, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2535}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:08:57.841 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7872", 'reported_seq': 7941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.086032+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:08:55.086032+0000', 'last_peered': '2024-09-15T21:08:55.086032+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:08:55.086032+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7872, 'log_dups_size': 0, 'ondisk_log_size': 7872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40047884, 'num_objects': 4889, 'num_object_clones': 0, 'num_object_copies': 19556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7870, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2415}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7703", 'reported_seq': 7763, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.745404+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:08:55.745404+0000', 'last_peered': '2024-09-15T21:08:55.745404+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:08:55.745404+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7703, 'log_dups_size': 0, 'ondisk_log_size': 7703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38898200, 'num_objects': 4749, 'num_object_clones': 0, 'num_object_copies': 18996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4749, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7700, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2417}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7658", 'reported_seq': 7739, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:52.800514+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:52.800514+0000', 'last_peered': '2024-09-15T21:08:52.800514+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:52.800514+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7658, 'log_dups_size': 0, 'ondisk_log_size': 7658, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40170764, 'num_objects': 4904, 'num_object_clones': 0, 'num_object_copies': 19616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4904, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7656, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2569}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7622", 'reported_seq': 7662, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.674748+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:08:55.674748+0000', 'last_peered': '2024-09-15T21:08:55.674748+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:08:55.674748+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7622, 'log_dups_size': 0, 'ondisk_log_size': 7622, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39361158, 'num_objects': 4805, 'num_object_clones': 0, 'num_object_copies': 19220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7621, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2549}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7629", 'reported_seq': 7708, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:52.866761+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:52.866761+0000', 'last_peered': '2024-09-15T21:08:52.866761+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:52.866761+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7629, 'log_dups_size': 0, 'ondisk_log_size': 7629, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39910022, 'num_objects': 4872, 'num_object_clones': 0, 'num_object_copies': 19488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7628, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2586}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:57.842 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:08:57.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7632", 'reported_seq': 7689, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.674131+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:08:55.674131+0000', 'last_peered': '2024-09-15T21:08:55.674131+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:08:55.674131+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7632, 'log_dups_size': 0, 'ondisk_log_size': 7632, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39318796, 'num_objects': 4800, 'num_object_clones': 0, 'num_object_copies': 19200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4800, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7630, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2521}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:08:57.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:08:57.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7703", 'reported_seq': 7786, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:52.831157+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:52.831157+0000', 'last_peered': '2024-09-15T21:08:52.831157+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:52.831157+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7703, 'log_dups_size': 0, 'ondisk_log_size': 7703, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40776972, 'num_objects': 4978, 'num_object_clones': 0, 'num_object_copies': 19912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4978, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7701, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2697}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:08:57.843 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:08:57.843 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7714", 'reported_seq': 7782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:55.120899+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:08:55.120899+0000', 'last_peered': '2024-09-15T21:08:55.120899+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:08:55.120899+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7714, 'log_dups_size': 0, 'ondisk_log_size': 7714, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39605516, 'num_objects': 4835, 'num_object_clones': 0, 'num_object_copies': 19340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7712, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2520}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:08:57.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:08:58.595 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:58.595 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33711/308208 objects degraded (10.938%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":404,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":77052,"data_bytes":631166766,"bytes_used":2832367616,"bytes_avail":383714689024,"bytes_total":386547056640,"degraded_objects":33711,"degraded_total":308208,"degraded_ratio":0.10937743342158543,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":62780,"recovering_keys_per_sec":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":231},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:08:58.629 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:08:58.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:08:59.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:08:59.237 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:08:59.257 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-09-15T21:08:58.041895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":631166766,"num_objects":77052,"num_object_clones":0,"num_object_copies":308208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117360,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123273,"ondisk_log_size":123273,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2765984,"kb_used_data":1183484,"kb_used_omap":14,"kb_used_meta":1582385,"kb_avail":374721376,"statfs":{"total":386547056640,"available":383714689024,"internally_reserved":0,"allocated":1211887616,"data_stored":1211227660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1620362802},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,4,9,8,10,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22732800,"num_objects":-2775,"num_object_clones":0,"num_object_copies":-11100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2775,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004817"},"pg_stats":[{"pgid":"1.f","version":"33'7569","reported_seq":7613,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.784019+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:08:55.784019+0000","last_peered":"2024-09-15T21:08:55.784019+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:08:55.784019+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7569,"log_dups_size":0,"ondisk_log_size":7569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39182336,"num_objects":4783,"num_object_clones":0,"num_object_copies":19132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7569,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2507}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7750","reported_seq":10042,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.068615+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:08:55.068615+0000","last_peered":"2024-09-15T21:08:55.068615+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:08:55.068615+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39335180,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1114,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3532}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7631","reported_seq":7711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.782987+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:57.782987+0000","last_peered":"2024-09-15T21:08:57.782987+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:57.782987+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39328414,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7630,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7636","reported_seq":7705,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965440+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:08:54.965440+0000","last_peered":"2024-09-15T21:08:54.965440+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:08:54.965440+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7636,"log_dups_size":0,"ondisk_log_size":7636,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39482636,"num_objects":4820,"num_object_clones":0,"num_object_copies":19280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7634,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7780","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.732372+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:08:55.732372+0000","last_peered":"2024-09-15T21:08:55.732372+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:08:55.732372+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7780,"log_dups_size":0,"ondisk_log_size":7780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39965964,"num_objects":4879,"num_object_clones":0,"num_object_copies":19516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2568}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7653","reported_seq":7698,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.458770+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:08:55.458770+0000","last_peered":"2024-09-15T21:08:55.458770+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:08:55.458770+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7653,"log_dups_size":0,"ondisk_log_size":7653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38690816,"num_objects":4723,"num_object_clones":0,"num_object_copies":18892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7653,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7871","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:54.965548+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:08:54.965548+0000","last_peered":"2024-09-15T21:08:54.965548+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:08:54.965548+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7871,"log_dups_size":0,"ondisk_log_size":7871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40146188,"num_objects":4901,"num_object_clones":0,"num_object_copies":19604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2535}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7872","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.086032+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:08:55.086032+0000","last_peered":"2024-09-15T21:08:55.086032+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:08:55.086032+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40047884,"num_objects":4889,"num_object_clones":0,"num_object_copies":19556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7870,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2415}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7703","reported_seq":7763,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.745404+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:08:55.745404+0000","last_peered":"2024-09-15T21:08:55.745404+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:08:55.745404+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7703,"log_dups_size":0,"ondisk_log_size":7703,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38898200,"num_objects":4749,"num_object_clones":0,"num_object_copies":18996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7700,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2417}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7749","reported_seq":7829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937128+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:57.937128+0000","last_peered":"2024-09-15T21:08:57.937128+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:57.937128+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425292,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7619","reported_seq":12281,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:08:55.756564+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:08:55.756564+0000","last_peered":"2024-09-15T21:08:55.756564+0000","last_clean":"2024-09-15T21:08:55.756564+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:08:55.756564+0000","last_undegraded":"2024-09-15T21:08:55.756564+0000","last_fullsized":"2024-09-15T21:08:55.756564+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7619,"log_dups_size":0,"ondisk_log_size":7619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38838272,"num_objects":4741,"num_object_clones":0,"num_object_copies":18964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1730,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7622","reported_seq":7662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674748+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:08:55.674748+0000","last_peered":"2024-09-15T21:08:55.674748+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:08:55.674748+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7622,"log_dups_size":0,"ondisk_log_size":7622,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7621,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2549}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7696","reported_seq":7775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937071+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:57.937071+0000","last_peered":"2024-09-15T21:08:57.937071+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:57.937071+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7632","reported_seq":7689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.674131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:08:55.674131+0000","last_peered":"2024-09-15T21:08:55.674131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:08:55.674131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7632,"log_dups_size":0,"ondisk_log_size":7632,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39318796,"num_objects":4800,"num_object_clones":0,"num_object_copies":19200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7630,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7776","reported_seq":7859,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.974215+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:57.974215+0000","last_peered":"2024-09-15T21:08:57.974215+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:57.974215+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40178956,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7714","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:55.120899+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:08:55.120899+0000","last_peered":"2024-09-15T21:08:55.120899+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:08:55.120899+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7714,"log_dups_size":0,"ondisk_log_size":7714,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39605516,"num_objects":4835,"num_object_clones":0,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2520}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":631166766,"num_objects":77052,"num_object_clones":0,"num_object_copies":308208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33711,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117360,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":9093120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1208479744,"data_stored":1208479744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123273,"ondisk_log_size":123273,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734316,"kb_used_data":309704,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93637524,"statfs":{"total":96636764160,"available":95884824576,"internally_reserved":0,"allocated":317136896,"data_stored":316971907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,2,1,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":307200,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":314572800,"data_stored":314407811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051644,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":549128,"kb_used_data":256504,"kb_used_omap":4,"kb_used_meta":292603,"kb_avail":93822712,"statfs":{"total":96636764160,"available":96074457088,"internally_reserved":0,"allocated":262660096,"data_stored":262495107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":299626191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247882,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742924,"kb_used_data":310076,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93628916,"statfs":{"total":96636764160,"available":95876009984,"internally_reserved":0,"allocated":317517824,"data_stored":317352835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,0,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":316567552,"data_stored":316567552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262361088,"data_stored":262361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313696256,"data_stored":313696256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":315854848,"data_stored":315854848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:02.259 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:02.886 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:02.887 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:02.917 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-15T21:09:02.043305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623662894,"num_objects":76136,"num_object_clones":0,"num_object_copies":304544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118276,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124189,"ondisk_log_size":124189,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2774032,"kb_used_data":1173504,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374713328,"statfs":{"total":386547056640,"available":383706447872,"internally_reserved":0,"allocated":1201668096,"data_stored":1201008140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,4,5,18,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27656192,"num_objects":-3376,"num_object_clones":0,"num_object_copies":-13504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004646"},"pg_stats":[{"pgid":"1.f","version":"33'7642","reported_seq":7688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.826387+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:00.826387+0000","last_peered":"2024-09-15T21:09:00.826387+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:00.826387+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7642,"log_dups_size":0,"ondisk_log_size":7642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584320,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7838","reported_seq":10214,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.194962+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:00.194962+0000","last_peered":"2024-09-15T21:09:00.194962+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:00.194962+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38614284,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7631","reported_seq":7711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.782987+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:57.782987+0000","last_peered":"2024-09-15T21:08:57.782987+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:57.782987+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39328414,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7630,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.060668+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:00.060668+0000","last_peered":"2024-09-15T21:09:00.060668+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:00.060668+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7856","reported_seq":7912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.835683+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:00.835683+0000","last_peered":"2024-09-15T21:09:00.835683+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:00.835683+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7725","reported_seq":7769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.804976+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:00.804976+0000","last_peered":"2024-09-15T21:09:00.804976+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:00.804976+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7725,"log_dups_size":0,"ondisk_log_size":7725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7943","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.018772+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:00.018772+0000","last_peered":"2024-09-15T21:09:00.018772+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:00.018772+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7960","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.156240+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:00.156240+0000","last_peered":"2024-09-15T21:09:00.156240+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:00.156240+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7775","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.670912+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:00.670912+0000","last_peered":"2024-09-15T21:09:00.670912+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:00.670912+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38308376,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7749","reported_seq":7829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937128+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:57.937128+0000","last_peered":"2024-09-15T21:08:57.937128+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:57.937128+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425292,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":12359,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:00.740221+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:00.740221+0000","last_peered":"2024-09-15T21:09:00.740221+0000","last_clean":"2024-09-15T21:09:00.740221+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:00.740221+0000","last_undegraded":"2024-09-15T21:09:00.740221+0000","last_fullsized":"2024-09-15T21:09:00.740221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38191104,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1809,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7697","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.646222+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:00.646222+0000","last_peered":"2024-09-15T21:09:00.646222+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:00.646222+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746758,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7696,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7696","reported_seq":7775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937071+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:57.937071+0000","last_peered":"2024-09-15T21:08:57.937071+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:57.937071+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7695","reported_seq":7751,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.802094+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:00.802094+0000","last_peered":"2024-09-15T21:09:00.802094+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:00.802094+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802700,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7776","reported_seq":7859,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.974215+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:57.974215+0000","last_peered":"2024-09-15T21:08:57.974215+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:57.974215+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40178956,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7795","reported_seq":7864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.062254+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:00.062254+0000","last_peered":"2024-09-15T21:09:00.062254+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:00.062254+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38941964,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7793,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":623662894,"num_objects":76136,"num_object_clones":0,"num_object_copies":304544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118276,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198198784,"data_stored":1198198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124189,"ondisk_log_size":124189,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737752,"kb_used_data":304840,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93634088,"statfs":{"total":96636764160,"available":95881306112,"internally_reserved":0,"allocated":312156160,"data_stored":311991171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":307200,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":314572800,"data_stored":314407811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554404,"kb_used_data":256212,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93817436,"statfs":{"total":96636764160,"available":96069054464,"internally_reserved":0,"allocated":262361088,"data_stored":262196099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305327823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742260,"kb_used_data":305252,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93629580,"statfs":{"total":96636764160,"available":95876689920,"internally_reserved":0,"allocated":312578048,"data_stored":312413059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311566336,"data_stored":311566336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262062080,"data_stored":262062080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313696256,"data_stored":313696256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310874112,"data_stored":310874112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:02.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:03.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:03.562 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:03.589 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-15T21:09:02.043305+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":623662894,"num_objects":76136,"num_object_clones":0,"num_object_copies":304544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118276,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124189,"ondisk_log_size":124189,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2774032,"kb_used_data":1173504,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374713328,"statfs":{"total":386547056640,"available":383706447872,"internally_reserved":0,"allocated":1201668096,"data_stored":1201008140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,7,4,5,18,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27656192,"num_objects":-3376,"num_object_clones":0,"num_object_copies":-13504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3376,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004646"},"pg_stats":[{"pgid":"1.f","version":"33'7642","reported_seq":7688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.826387+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:00.826387+0000","last_peered":"2024-09-15T21:09:00.826387+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:00.826387+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7642,"log_dups_size":0,"ondisk_log_size":7642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584320,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7838","reported_seq":10214,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.194962+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:00.194962+0000","last_peered":"2024-09-15T21:09:00.194962+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:00.194962+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38614284,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7631","reported_seq":7711,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.782987+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:08:57.782987+0000","last_peered":"2024-09-15T21:08:57.782987+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:08:57.782987+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39328414,"num_objects":4802,"num_object_clones":0,"num_object_copies":19208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7630,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.060668+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:00.060668+0000","last_peered":"2024-09-15T21:09:00.060668+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:00.060668+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7856","reported_seq":7912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.835683+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:00.835683+0000","last_peered":"2024-09-15T21:09:00.835683+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:00.835683+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7725","reported_seq":7769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.804976+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:00.804976+0000","last_peered":"2024-09-15T21:09:00.804976+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:00.804976+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7725,"log_dups_size":0,"ondisk_log_size":7725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7943","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.018772+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:00.018772+0000","last_peered":"2024-09-15T21:09:00.018772+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:00.018772+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7960","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.156240+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:00.156240+0000","last_peered":"2024-09-15T21:09:00.156240+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:00.156240+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7775","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.670912+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:00.670912+0000","last_peered":"2024-09-15T21:09:00.670912+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:00.670912+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38308376,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7749","reported_seq":7829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937128+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:08:57.937128+0000","last_peered":"2024-09-15T21:08:57.937128+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:08:57.937128+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7749,"log_dups_size":0,"ondisk_log_size":7749,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39425292,"num_objects":4813,"num_object_clones":0,"num_object_copies":19252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2478}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":12359,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:00.740221+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:00.740221+0000","last_peered":"2024-09-15T21:09:00.740221+0000","last_clean":"2024-09-15T21:09:00.740221+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:00.740221+0000","last_undegraded":"2024-09-15T21:09:00.740221+0000","last_fullsized":"2024-09-15T21:09:00.740221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38191104,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1809,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7697","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.646222+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:00.646222+0000","last_peered":"2024-09-15T21:09:00.646222+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:00.646222+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746758,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7696,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7696","reported_seq":7775,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.937071+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:08:57.937071+0000","last_peered":"2024-09-15T21:08:57.937071+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:08:57.937071+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7696,"log_dups_size":0,"ondisk_log_size":7696,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39361158,"num_objects":4805,"num_object_clones":0,"num_object_copies":19220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7695,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7695","reported_seq":7751,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.802094+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:00.802094+0000","last_peered":"2024-09-15T21:09:00.802094+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:00.802094+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802700,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7776","reported_seq":7859,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:08:57.974215+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:08:57.974215+0000","last_peered":"2024-09-15T21:08:57.974215+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:08:57.974215+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7776,"log_dups_size":0,"ondisk_log_size":7776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40178956,"num_objects":4905,"num_object_clones":0,"num_object_copies":19620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2624}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7795","reported_seq":7864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.062254+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:00.062254+0000","last_peered":"2024-09-15T21:09:00.062254+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:00.062254+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38941964,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7793,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":623662894,"num_objects":76136,"num_object_clones":0,"num_object_copies":304544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118276,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1198198784,"data_stored":1198198784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124189,"ondisk_log_size":124189,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737752,"kb_used_data":304840,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93634088,"statfs":{"total":96636764160,"available":95881306112,"internally_reserved":0,"allocated":312156160,"data_stored":311991171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739616,"kb_used_data":307200,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93632224,"statfs":{"total":96636764160,"available":95879397376,"internally_reserved":0,"allocated":314572800,"data_stored":314407811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051645,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":554404,"kb_used_data":256212,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93817436,"statfs":{"total":96636764160,"available":96069054464,"internally_reserved":0,"allocated":262361088,"data_stored":262196099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305327823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742260,"kb_used_data":305252,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93629580,"statfs":{"total":96636764160,"available":95876689920,"internally_reserved":0,"allocated":312578048,"data_stored":312413059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311566336,"data_stored":311566336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":262062080,"data_stored":262062080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":313696256,"data_stored":313696256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310874112,"data_stored":310874112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7642", 'reported_seq': 7688, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.826387+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:00.826387+0000', 'last_peered': '2024-09-15T21:09:00.826387+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:00.826387+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7642, 'log_dups_size': 0, 'ondisk_log_size': 7642, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38584320, 'num_objects': 4710, 'num_object_clones': 0, 'num_object_copies': 18840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4710, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7642, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2434}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7838", 'reported_seq': 10214, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.194962+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:00.194962+0000', 'last_peered': '2024-09-15T21:09:00.194962+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:00.194962+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7838, 'log_dups_size': 0, 'ondisk_log_size': 7838, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38614284, 'num_objects': 4714, 'num_object_clones': 0, 'num_object_copies': 18856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1228, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4714, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7836, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1156, 'num_bytes_recovered': 9437184, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7631", 'reported_seq': 7711, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:57.782987+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:08:57.782987+0000', 'last_peered': '2024-09-15T21:08:57.782987+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:08:57.782987+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7631, 'log_dups_size': 0, 'ondisk_log_size': 7631, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39328414, 'num_objects': 4802, 'num_object_clones': 0, 'num_object_copies': 19208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4802, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7630, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2544}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:03.590 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7713", 'reported_seq': 7782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.060668+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:00.060668+0000', 'last_peered': '2024-09-15T21:09:00.060668+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:00.060668+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7713, 'log_dups_size': 0, 'ondisk_log_size': 7713, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38851852, 'num_objects': 4743, 'num_object_clones': 0, 'num_object_copies': 18972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7711, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2409}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7856", 'reported_seq': 7912, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.835683+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:00.835683+0000', 'last_peered': '2024-09-15T21:09:00.835683+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:00.835683+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7856, 'log_dups_size': 0, 'ondisk_log_size': 7856, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39343372, 'num_objects': 4803, 'num_object_clones': 0, 'num_object_copies': 19212, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4803, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7854, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2492}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7725", 'reported_seq': 7769, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.804976+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:00.804976+0000', 'last_peered': '2024-09-15T21:09:00.804976+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:00.804976+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7725, 'log_dups_size': 0, 'ondisk_log_size': 7725, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38100992, 'num_objects': 4651, 'num_object_clones': 0, 'num_object_copies': 18604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7725, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2335}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'7943", 'reported_seq': 8013, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.018772+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:00.018772+0000', 'last_peered': '2024-09-15T21:09:00.018772+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:00.018772+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39556364, 'num_objects': 4829, 'num_object_clones': 0, 'num_object_copies': 19316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7941, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2463}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:03.591 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'7960", 'reported_seq': 8029, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.156240+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:00.156240+0000', 'last_peered': '2024-09-15T21:09:00.156240+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:00.156240+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7960, 'log_dups_size': 0, 'ondisk_log_size': 7960, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39326988, 'num_objects': 4801, 'num_object_clones': 0, 'num_object_copies': 19204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7958, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2327}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7775", 'reported_seq': 7835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.670912+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:00.670912+0000', 'last_peered': '2024-09-15T21:09:00.670912+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:00.670912+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7775, 'log_dups_size': 0, 'ondisk_log_size': 7775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38308376, 'num_objects': 4677, 'num_object_clones': 0, 'num_object_copies': 18708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4677, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7772, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2345}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7749", 'reported_seq': 7829, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:57.937128+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:08:57.937128+0000', 'last_peered': '2024-09-15T21:08:57.937128+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:08:57.937128+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7749, 'log_dups_size': 0, 'ondisk_log_size': 7749, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39425292, 'num_objects': 4813, 'num_object_clones': 0, 'num_object_copies': 19252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7747, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2478}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7697", 'reported_seq': 7737, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.646222+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:00.646222+0000', 'last_peered': '2024-09-15T21:09:00.646222+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:00.646222+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7697, 'log_dups_size': 0, 'ondisk_log_size': 7697, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38746758, 'num_objects': 4730, 'num_object_clones': 0, 'num_object_copies': 18920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7696, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:03.592 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7696", 'reported_seq': 7775, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:57.937071+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:08:57.937071+0000', 'last_peered': '2024-09-15T21:08:57.937071+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:08:57.937071+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7696, 'log_dups_size': 0, 'ondisk_log_size': 7696, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39361158, 'num_objects': 4805, 'num_object_clones': 0, 'num_object_copies': 19220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4805, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7695, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2519}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7695", 'reported_seq': 7751, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.802094+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:00.802094+0000', 'last_peered': '2024-09-15T21:09:00.802094+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:00.802094+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7695, 'log_dups_size': 0, 'ondisk_log_size': 7695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38802700, 'num_objects': 4737, 'num_object_clones': 0, 'num_object_copies': 18948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4737, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7693, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7776", 'reported_seq': 7859, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:08:57.974215+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:08:57.974215+0000', 'last_peered': '2024-09-15T21:08:57.974215+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:08:57.974215+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7776, 'log_dups_size': 0, 'ondisk_log_size': 7776, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 40178956, 'num_objects': 4905, 'num_object_clones': 0, 'num_object_copies': 19620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4905, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:03.593 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7795", 'reported_seq': 7864, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:00.062254+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:00.062254+0000', 'last_peered': '2024-09-15T21:09:00.062254+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:00.062254+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7795, 'log_dups_size': 0, 'ondisk_log_size': 7795, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38941964, 'num_objects': 4754, 'num_object_clones': 0, 'num_object_copies': 19016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4754, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7793, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2439}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:03.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:04.395 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:04.395 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33669/304544 objects degraded (11.056%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":76136,"data_bytes":623662894,"bytes_used":2840608768,"bytes_avail":383706447872,"bytes_total":386547056640,"degraded_objects":33669,"degraded_total":304544,"degraded_ratio":0.11055545339918041,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":91441,"recovering_keys_per_sec":0,"num_objects_recovered":134,"num_bytes_recovered":1097728,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":281},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:04.420 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:04.420 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:05.080 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:05.081 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:05.107 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-15T21:09:04.044052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":621246254,"num_objects":75841,"num_object_clones":0,"num_object_copies":303364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118571,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124484,"ondisk_log_size":124484,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2777136,"kb_used_data":1168304,"kb_used_omap":14,"kb_used_meta":1608753,"kb_avail":374710224,"statfs":{"total":386547056640,"available":383703269376,"internally_reserved":0,"allocated":1196343296,"data_stored":1195683340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1647363634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,6,10,8,18],"upper_bound":128},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22282240,"num_objects":-2720,"num_object_clones":0,"num_object_copies":-10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2720,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":696320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004625"},"pg_stats":[{"pgid":"1.f","version":"33'7642","reported_seq":7688,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.826387+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:00.826387+0000","last_peered":"2024-09-15T21:09:00.826387+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:00.826387+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7642,"log_dups_size":0,"ondisk_log_size":7642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38584320,"num_objects":4710,"num_object_clones":0,"num_object_copies":18840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7642,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2434}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7838","reported_seq":10214,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.194962+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:00.194962+0000","last_peered":"2024-09-15T21:09:00.194962+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:00.194962+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7838,"log_dups_size":0,"ondisk_log_size":7838,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38614284,"num_objects":4714,"num_object_clones":0,"num_object_copies":18856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7836,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7700","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.699683+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:02.699683+0000","last_peered":"2024-09-15T21:09:02.699683+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:02.699683+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38763166,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7699,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2475}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7713","reported_seq":7782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.060668+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:00.060668+0000","last_peered":"2024-09-15T21:09:00.060668+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:00.060668+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7711,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2409}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7856","reported_seq":7912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.835683+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:00.835683+0000","last_peered":"2024-09-15T21:09:00.835683+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:00.835683+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7856,"log_dups_size":0,"ondisk_log_size":7856,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39343372,"num_objects":4803,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2492}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7725","reported_seq":7769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.804976+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:00.804976+0000","last_peered":"2024-09-15T21:09:00.804976+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:00.804976+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7725,"log_dups_size":0,"ondisk_log_size":7725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38100992,"num_objects":4651,"num_object_clones":0,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7725,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2335}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'7943","reported_seq":8013,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.018772+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:00.018772+0000","last_peered":"2024-09-15T21:09:00.018772+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:00.018772+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39556364,"num_objects":4829,"num_object_clones":0,"num_object_copies":19316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7941,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'7960","reported_seq":8029,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.156240+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:00.156240+0000","last_peered":"2024-09-15T21:09:00.156240+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:00.156240+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7960,"log_dups_size":0,"ondisk_log_size":7960,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39326988,"num_objects":4801,"num_object_clones":0,"num_object_copies":19204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7958,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2327}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7775","reported_seq":7835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.670912+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:00.670912+0000","last_peered":"2024-09-15T21:09:00.670912+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:00.670912+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38308376,"num_objects":4677,"num_object_clones":0,"num_object_copies":18708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7772,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2345}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7819","reported_seq":7901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.762682+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:02.762682+0000","last_peered":"2024-09-15T21:09:02.762682+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:02.762682+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7698","reported_seq":12359,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:00.740221+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:00.740221+0000","last_peered":"2024-09-15T21:09:00.740221+0000","last_clean":"2024-09-15T21:09:00.740221+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:00.740221+0000","last_undegraded":"2024-09-15T21:09:00.740221+0000","last_fullsized":"2024-09-15T21:09:00.740221+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38191104,"num_objects":4662,"num_object_clones":0,"num_object_copies":18648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1809,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7697","reported_seq":7737,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.646222+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:00.646222+0000","last_peered":"2024-09-15T21:09:00.646222+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:00.646222+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7697,"log_dups_size":0,"ondisk_log_size":7697,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38746758,"num_objects":4730,"num_object_clones":0,"num_object_copies":18920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7696,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7775","reported_seq":7855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.945042+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:02.945042+0000","last_peered":"2024-09-15T21:09:02.945042+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:02.945042+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713990,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7695","reported_seq":7751,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.802094+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:00.802094+0000","last_peered":"2024-09-15T21:09:00.802094+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:00.802094+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7695,"log_dups_size":0,"ondisk_log_size":7695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38802700,"num_objects":4737,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7693,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7853","reported_seq":7936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.657254+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:02.657254+0000","last_peered":"2024-09-15T21:09:02.657254+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:02.657254+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39548172,"num_objects":4828,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7795","reported_seq":7864,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:00.062254+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:00.062254+0000","last_peered":"2024-09-15T21:09:00.062254+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:00.062254+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7795,"log_dups_size":0,"ondisk_log_size":7795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38941964,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7793,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2439}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":621246254,"num_objects":75841,"num_object_clones":0,"num_object_copies":303364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75841,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":118571,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1178,"num_bytes_recovered":9437184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1192919040,"data_stored":1192919040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124484,"ondisk_log_size":124484,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737752,"kb_used_data":304840,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93634088,"statfs":{"total":96636764160,"available":95881306112,"internally_reserved":0,"allocated":312156160,"data_stored":311991171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738800,"kb_used_data":302240,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93633040,"statfs":{"total":96636764160,"available":95880232960,"internally_reserved":0,"allocated":309493760,"data_stored":309328771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558324,"kb_used_data":255972,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93813516,"statfs":{"total":96636764160,"available":96065040384,"internally_reserved":0,"allocated":262115328,"data_stored":261950339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742260,"kb_used_data":305252,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93629580,"statfs":{"total":96636764160,"available":95876689920,"internally_reserved":0,"allocated":312578048,"data_stored":312413059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311566336,"data_stored":311566336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261799936,"data_stored":261799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308678656,"data_stored":308678656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310874112,"data_stored":310874112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:08.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:08.715 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:08.715 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:08.738 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-15T21:09:08.045754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":613660462,"num_objects":74915,"num_object_clones":0,"num_object_copies":299660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119497,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125410,"ondisk_log_size":125410,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759072,"kb_used_data":1158552,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374728288,"statfs":{"total":386547056640,"available":383721766912,"internally_reserved":0,"allocated":1186357248,"data_stored":1185697292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,6,21,11,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19963904,"num_objects":-2437,"num_object_clones":0,"num_object_copies":-9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"33'7729","reported_seq":7774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.868613+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:05.868613+0000","last_peered":"2024-09-15T21:09:05.868613+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:05.868613+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37871616,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7909","reported_seq":10381,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.243249+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:05.243249+0000","last_peered":"2024-09-15T21:09:05.243249+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:05.243249+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032652,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1204,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7700","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.699683+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:02.699683+0000","last_peered":"2024-09-15T21:09:02.699683+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:02.699683+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38763166,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7699,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2475}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7798","reported_seq":7867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.188704+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:05.188704+0000","last_peered":"2024-09-15T21:09:05.188704+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:05.188704+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7924","reported_seq":7980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.718397+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:05.718397+0000","last_peered":"2024-09-15T21:09:05.718397+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:05.718397+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2424}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7799","reported_seq":7844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.768769+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:05.768769+0000","last_peered":"2024-09-15T21:09:05.768769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:05.768769+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8017","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.143153+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:05.143153+0000","last_peered":"2024-09-15T21:09:05.143153+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:05.143153+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950156,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8046","reported_seq":8115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.208895+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:05.208895+0000","last_peered":"2024-09-15T21:09:05.208895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:05.208895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622476,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7866","reported_seq":7926,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.873015+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:05.873015+0000","last_peered":"2024-09-15T21:09:05.873015+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:05.873015+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37562904,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7863,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7819","reported_seq":7901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.762682+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:02.762682+0000","last_peered":"2024-09-15T21:09:02.762682+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:02.762682+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7765","reported_seq":12427,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:05.840287+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:05.840287+0000","last_peered":"2024-09-15T21:09:05.840287+0000","last_clean":"2024-09-15T21:09:05.840287+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:05.840287+0000","last_undegraded":"2024-09-15T21:09:05.840287+0000","last_fullsized":"2024-09-15T21:09:05.840287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37642240,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1876,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7775","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.727717+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:05.727717+0000","last_peered":"2024-09-15T21:09:05.727717+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:05.727717+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38107782,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2396}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7775","reported_seq":7855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.945042+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:02.945042+0000","last_peered":"2024-09-15T21:09:02.945042+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:02.945042+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713990,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7762","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.674679+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:05.674679+0000","last_peered":"2024-09-15T21:09:05.674679+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:05.674679+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38253836,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2391}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7853","reported_seq":7936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.657254+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:02.657254+0000","last_peered":"2024-09-15T21:09:02.657254+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:02.657254+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39548172,"num_objects":4828,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7873","reported_seq":7942,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.103723+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:05.103723+0000","last_peered":"2024-09-15T21:09:05.103723+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:05.103723+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302988,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":613660462,"num_objects":74915,"num_object_clones":0,"num_object_copies":299660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119497,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1182920704,"data_stored":1182920704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125410,"ondisk_log_size":125410,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736972,"kb_used_data":299896,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93634868,"statfs":{"total":96636764160,"available":95882104832,"internally_reserved":0,"allocated":307093504,"data_stored":306928515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,1,7,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738800,"kb_used_data":302240,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93633040,"statfs":{"total":96636764160,"available":95880232960,"internally_reserved":0,"allocated":309493760,"data_stored":309328771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558324,"kb_used_data":255972,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93813516,"statfs":{"total":96636764160,"available":96065040384,"internally_reserved":0,"allocated":262115328,"data_stored":261950339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724976,"kb_used_data":300444,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93646864,"statfs":{"total":96636764160,"available":95894388736,"internally_reserved":0,"allocated":307654656,"data_stored":307489667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":434695886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,5,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306561024,"data_stored":306561024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261799936,"data_stored":261799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308678656,"data_stored":308678656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305881088,"data_stored":305881088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:08.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:09.354 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:09.355 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:09.382 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-15T21:09:08.045754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":613660462,"num_objects":74915,"num_object_clones":0,"num_object_copies":299660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119497,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125410,"ondisk_log_size":125410,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759072,"kb_used_data":1158552,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374728288,"statfs":{"total":386547056640,"available":383721766912,"internally_reserved":0,"allocated":1186357248,"data_stored":1185697292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,4,6,21,11,12,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19963904,"num_objects":-2437,"num_object_clones":0,"num_object_copies":-9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"33'7729","reported_seq":7774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.868613+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:05.868613+0000","last_peered":"2024-09-15T21:09:05.868613+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:05.868613+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37871616,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7909","reported_seq":10381,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.243249+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:05.243249+0000","last_peered":"2024-09-15T21:09:05.243249+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:05.243249+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032652,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1204,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7700","reported_seq":7780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.699683+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:02.699683+0000","last_peered":"2024-09-15T21:09:02.699683+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:02.699683+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38763166,"num_objects":4733,"num_object_clones":0,"num_object_copies":18932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4733,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7699,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2475}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7798","reported_seq":7867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.188704+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:05.188704+0000","last_peered":"2024-09-15T21:09:05.188704+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:05.188704+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7924","reported_seq":7980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.718397+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:05.718397+0000","last_peered":"2024-09-15T21:09:05.718397+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:05.718397+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2424}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7799","reported_seq":7844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.768769+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:05.768769+0000","last_peered":"2024-09-15T21:09:05.768769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:05.768769+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8017","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.143153+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:05.143153+0000","last_peered":"2024-09-15T21:09:05.143153+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:05.143153+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950156,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8046","reported_seq":8115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.208895+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:05.208895+0000","last_peered":"2024-09-15T21:09:05.208895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:05.208895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622476,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7866","reported_seq":7926,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.873015+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:05.873015+0000","last_peered":"2024-09-15T21:09:05.873015+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:05.873015+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37562904,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7863,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7819","reported_seq":7901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.762682+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:02.762682+0000","last_peered":"2024-09-15T21:09:02.762682+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:02.762682+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7819,"log_dups_size":0,"ondisk_log_size":7819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38851852,"num_objects":4743,"num_object_clones":0,"num_object_copies":18972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7765","reported_seq":12427,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:05.840287+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:05.840287+0000","last_peered":"2024-09-15T21:09:05.840287+0000","last_clean":"2024-09-15T21:09:05.840287+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:05.840287+0000","last_undegraded":"2024-09-15T21:09:05.840287+0000","last_fullsized":"2024-09-15T21:09:05.840287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37642240,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1876,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7775","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.727717+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:05.727717+0000","last_peered":"2024-09-15T21:09:05.727717+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:05.727717+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38107782,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2396}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7775","reported_seq":7855,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.945042+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:02.945042+0000","last_peered":"2024-09-15T21:09:02.945042+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:02.945042+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38713990,"num_objects":4726,"num_object_clones":0,"num_object_copies":18904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7762","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.674679+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:05.674679+0000","last_peered":"2024-09-15T21:09:05.674679+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:05.674679+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38253836,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2391}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7853","reported_seq":7936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:02.657254+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:02.657254+0000","last_peered":"2024-09-15T21:09:02.657254+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:02.657254+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7853,"log_dups_size":0,"ondisk_log_size":7853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39548172,"num_objects":4828,"num_object_clones":0,"num_object_copies":19312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7873","reported_seq":7942,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.103723+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:05.103723+0000","last_peered":"2024-09-15T21:09:05.103723+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:05.103723+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302988,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":613660462,"num_objects":74915,"num_object_clones":0,"num_object_copies":299660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74915,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119497,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1182920704,"data_stored":1182920704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125410,"ondisk_log_size":125410,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736972,"kb_used_data":299896,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93634868,"statfs":{"total":96636764160,"available":95882104832,"internally_reserved":0,"allocated":307093504,"data_stored":306928515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,1,7,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738800,"kb_used_data":302240,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93633040,"statfs":{"total":96636764160,"available":95880232960,"internally_reserved":0,"allocated":309493760,"data_stored":309328771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051646,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558324,"kb_used_data":255972,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93813516,"statfs":{"total":96636764160,"available":96065040384,"internally_reserved":0,"allocated":262115328,"data_stored":261950339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724976,"kb_used_data":300444,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93646864,"statfs":{"total":96636764160,"available":95894388736,"internally_reserved":0,"allocated":307654656,"data_stored":307489667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":434695886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,5,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306561024,"data_stored":306561024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261799936,"data_stored":261799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308678656,"data_stored":308678656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305881088,"data_stored":305881088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7729", 'reported_seq': 7774, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.868613+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:05.868613+0000', 'last_peered': '2024-09-15T21:09:05.868613+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:05.868613+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7729, 'log_dups_size': 0, 'ondisk_log_size': 7729, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37871616, 'num_objects': 4623, 'num_object_clones': 0, 'num_object_copies': 18492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4623, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7729, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2347}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7909", 'reported_seq': 10381, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.243249+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:05.243249+0000', 'last_peered': '2024-09-15T21:09:05.243249+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:05.243249+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7909, 'log_dups_size': 0, 'ondisk_log_size': 7909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38032652, 'num_objects': 4643, 'num_object_clones': 0, 'num_object_copies': 18572, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1180, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4643, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7907, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1204, 'num_bytes_recovered': 9830400, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:09.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7700", 'reported_seq': 7780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:02.699683+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:02.699683+0000', 'last_peered': '2024-09-15T21:09:02.699683+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:02.699683+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7700, 'log_dups_size': 0, 'ondisk_log_size': 7700, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38763166, 'num_objects': 4733, 'num_object_clones': 0, 'num_object_copies': 18932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4733, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7699, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2475}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7798", 'reported_seq': 7867, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.188704+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:05.188704+0000', 'last_peered': '2024-09-15T21:09:05.188704+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:05.188704+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7798, 'log_dups_size': 0, 'ondisk_log_size': 7798, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38155532, 'num_objects': 4658, 'num_object_clones': 0, 'num_object_copies': 18632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7796, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2324}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'7924", 'reported_seq': 7980, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.718397+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:05.718397+0000', 'last_peered': '2024-09-15T21:09:05.718397+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:05.718397+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7924, 'log_dups_size': 0, 'ondisk_log_size': 7924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38786316, 'num_objects': 4735, 'num_object_clones': 0, 'num_object_copies': 18940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7922, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2424}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7799", 'reported_seq': 7844, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.768769+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:05.768769+0000', 'last_peered': '2024-09-15T21:09:05.768769+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:05.768769+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7799, 'log_dups_size': 0, 'ondisk_log_size': 7799, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37494784, 'num_objects': 4577, 'num_object_clones': 0, 'num_object_copies': 18308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7799, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8017", 'reported_seq': 8086, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.143153+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:05.143153+0000', 'last_peered': '2024-09-15T21:09:05.143153+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:05.143153+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8017, 'log_dups_size': 0, 'ondisk_log_size': 8017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38950156, 'num_objects': 4755, 'num_object_clones': 0, 'num_object_copies': 19020, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4755, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8015, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2389}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:09.384 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8046", 'reported_seq': 8115, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.208895+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:05.208895+0000', 'last_peered': '2024-09-15T21:09:05.208895+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:05.208895+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8046, 'log_dups_size': 0, 'ondisk_log_size': 8046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38622476, 'num_objects': 4715, 'num_object_clones': 0, 'num_object_copies': 18860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8044, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7866", 'reported_seq': 7926, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.873015+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:05.873015+0000', 'last_peered': '2024-09-15T21:09:05.873015+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:05.873015+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7866, 'log_dups_size': 0, 'ondisk_log_size': 7866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37562904, 'num_objects': 4586, 'num_object_clones': 0, 'num_object_copies': 18344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4586, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7863, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2254}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7819", 'reported_seq': 7901, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:02.762682+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:02.762682+0000', 'last_peered': '2024-09-15T21:09:02.762682+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:02.762682+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7819, 'log_dups_size': 0, 'ondisk_log_size': 7819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38851852, 'num_objects': 4743, 'num_object_clones': 0, 'num_object_copies': 18972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7817, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2408}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7775", 'reported_seq': 7815, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.727717+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:05.727717+0000', 'last_peered': '2024-09-15T21:09:05.727717+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:05.727717+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7775, 'log_dups_size': 0, 'ondisk_log_size': 7775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38107782, 'num_objects': 4652, 'num_object_clones': 0, 'num_object_copies': 18608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4652, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7775", 'reported_seq': 7855, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:02.945042+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:02.945042+0000', 'last_peered': '2024-09-15T21:09:02.945042+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:02.945042+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7775, 'log_dups_size': 0, 'ondisk_log_size': 7775, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38713990, 'num_objects': 4726, 'num_object_clones': 0, 'num_object_copies': 18904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4726, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7774, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2440}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:09.385 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:09.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7762", 'reported_seq': 7819, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.674679+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:05.674679+0000', 'last_peered': '2024-09-15T21:09:05.674679+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:05.674679+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7762, 'log_dups_size': 0, 'ondisk_log_size': 7762, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38253836, 'num_objects': 4670, 'num_object_clones': 0, 'num_object_copies': 18680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4670, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7760, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2391}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:09.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:09.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7853", 'reported_seq': 7936, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:02.657254+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:02.657254+0000', 'last_peered': '2024-09-15T21:09:02.657254+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:02.657254+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7853, 'log_dups_size': 0, 'ondisk_log_size': 7853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 39548172, 'num_objects': 4828, 'num_object_clones': 0, 'num_object_copies': 19312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7851, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:09.386 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:09.386 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7873", 'reported_seq': 7942, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:05.103723+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:05.103723+0000', 'last_peered': '2024-09-15T21:09:05.103723+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:05.103723+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7873, 'log_dups_size': 0, 'ondisk_log_size': 7873, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38302988, 'num_objects': 4676, 'num_object_clones': 0, 'num_object_copies': 18704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7871, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2361}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:09.386 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:10.090 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:10.090 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33621/299660 objects degraded (11.220%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":415,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":74915,"data_bytes":613660462,"bytes_used":2825289728,"bytes_avail":383721766912,"bytes_total":386547056640,"degraded_objects":33621,"degraded_total":299660,"degraded_ratio":0.11219715677768137,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":61415,"recovering_keys_per_sec":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:10.119 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:10.119 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:10.735 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:10.736 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:10.764 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-15T21:09:10.046557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":611235630,"num_objects":74619,"num_object_clones":0,"num_object_copies":298476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119793,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125706,"ondisk_log_size":125706,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2745740,"kb_used_data":1153540,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374741620,"statfs":{"total":386547056640,"available":383735418880,"internally_reserved":0,"allocated":1181224960,"data_stored":1180565004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1630324274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,7,8,18,14,10,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19931136,"num_objects":-2433,"num_object_clones":0,"num_object_copies":-9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-90,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2433,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":737280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004662"},"pg_stats":[{"pgid":"1.f","version":"33'7729","reported_seq":7774,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.868613+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:05.868613+0000","last_peered":"2024-09-15T21:09:05.868613+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:05.868613+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37871616,"num_objects":4623,"num_object_clones":0,"num_object_copies":18492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7909","reported_seq":10381,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.243249+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:05.243249+0000","last_peered":"2024-09-15T21:09:05.243249+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:05.243249+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7909,"log_dups_size":0,"ondisk_log_size":7909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38032652,"num_objects":4643,"num_object_clones":0,"num_object_copies":18572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1204,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7783","reported_seq":7863,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:07.931511+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:07.931511+0000","last_peered":"2024-09-15T21:09:07.931511+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:07.931511+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7783,"log_dups_size":0,"ondisk_log_size":7783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38083230,"num_objects":4650,"num_object_clones":0,"num_object_copies":18600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7782,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2392}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7798","reported_seq":7867,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.188704+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:05.188704+0000","last_peered":"2024-09-15T21:09:05.188704+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:05.188704+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7798,"log_dups_size":0,"ondisk_log_size":7798,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38155532,"num_objects":4658,"num_object_clones":0,"num_object_copies":18632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7796,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'7924","reported_seq":7980,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.718397+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:05.718397+0000","last_peered":"2024-09-15T21:09:05.718397+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:05.718397+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7924,"log_dups_size":0,"ondisk_log_size":7924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38786316,"num_objects":4735,"num_object_clones":0,"num_object_copies":18940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7922,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2424}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7799","reported_seq":7844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.768769+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:05.768769+0000","last_peered":"2024-09-15T21:09:05.768769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:05.768769+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7799,"log_dups_size":0,"ondisk_log_size":7799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37494784,"num_objects":4577,"num_object_clones":0,"num_object_copies":18308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7799,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8017","reported_seq":8086,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.143153+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:05.143153+0000","last_peered":"2024-09-15T21:09:05.143153+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:05.143153+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8017,"log_dups_size":0,"ondisk_log_size":8017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38950156,"num_objects":4755,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8015,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8046","reported_seq":8115,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.208895+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:05.208895+0000","last_peered":"2024-09-15T21:09:05.208895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:05.208895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38622476,"num_objects":4715,"num_object_clones":0,"num_object_copies":18860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8044,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2241}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7866","reported_seq":7926,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.873015+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:05.873015+0000","last_peered":"2024-09-15T21:09:05.873015+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:05.873015+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7866,"log_dups_size":0,"ondisk_log_size":7866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37562904,"num_objects":4586,"num_object_clones":0,"num_object_copies":18344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7863,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2254}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7902","reported_seq":7984,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:08.002197+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:08.002197+0000","last_peered":"2024-09-15T21:09:08.002197+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:08.002197+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7902,"log_dups_size":0,"ondisk_log_size":7902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38171916,"num_objects":4660,"num_object_clones":0,"num_object_copies":18640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7900,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2325}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7765","reported_seq":12427,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:05.840287+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:05.840287+0000","last_peered":"2024-09-15T21:09:05.840287+0000","last_clean":"2024-09-15T21:09:05.840287+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:05.840287+0000","last_undegraded":"2024-09-15T21:09:05.840287+0000","last_fullsized":"2024-09-15T21:09:05.840287+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37642240,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1876,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7775","reported_seq":7815,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.727717+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:05.727717+0000","last_peered":"2024-09-15T21:09:05.727717+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:05.727717+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38107782,"num_objects":4652,"num_object_clones":0,"num_object_copies":18608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2396}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7831","reported_seq":7911,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:07.996092+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:07.996092+0000","last_peered":"2024-09-15T21:09:07.996092+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:07.996092+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7831,"log_dups_size":0,"ondisk_log_size":7831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38255238,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7830,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2384}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7762","reported_seq":7819,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.674679+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:05.674679+0000","last_peered":"2024-09-15T21:09:05.674679+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:05.674679+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38253836,"num_objects":4670,"num_object_clones":0,"num_object_copies":18680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7760,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2391}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7927","reported_seq":8010,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:07.888295+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:07.888295+0000","last_peered":"2024-09-15T21:09:07.888295+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:07.888295+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7927,"log_dups_size":0,"ondisk_log_size":7927,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38941964,"num_objects":4754,"num_object_clones":0,"num_object_copies":19016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7925,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2473}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7873","reported_seq":7942,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:05.103723+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:05.103723+0000","last_peered":"2024-09-15T21:09:05.103723+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:05.103723+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38302988,"num_objects":4676,"num_object_clones":0,"num_object_copies":18704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2361}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":611235630,"num_objects":74619,"num_object_clones":0,"num_object_copies":298476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33621,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74619,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":119793,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":9830400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1177677824,"data_stored":1177677824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125706,"ondisk_log_size":125706,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736972,"kb_used_data":299896,"kb_used_omap":4,"kb_used_meta":437051,"kb_avail":93634868,"statfs":{"total":96636764160,"available":95882104832,"internally_reserved":0,"allocated":307093504,"data_stored":306928515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":447540944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,1,7,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721512,"kb_used_data":297432,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93650328,"statfs":{"total":96636764160,"available":95897935872,"internally_reserved":0,"allocated":304570368,"data_stored":304405379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051647,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":562280,"kb_used_data":255768,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93809560,"statfs":{"total":96636764160,"available":96060989440,"internally_reserved":0,"allocated":261906432,"data_stored":261741443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724976,"kb_used_data":300444,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93646864,"statfs":{"total":96636764160,"available":95894388736,"internally_reserved":0,"allocated":307654656,"data_stored":307489667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":434695886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,5,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":306561024,"data_stored":306561024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261599232,"data_stored":261599232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":303636480,"data_stored":303636480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305881088,"data_stored":305881088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:13.766 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:14.325 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:14.327 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:14.359 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-15T21:09:14.048357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601118510,"num_objects":73384,"num_object_clones":0,"num_object_copies":293536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121028,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126941,"ondisk_log_size":126941,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2734932,"kb_used_data":1138568,"kb_used_omap":14,"kb_used_meta":1596273,"kb_avail":374752428,"statfs":{"total":386547056640,"available":383746486272,"internally_reserved":0,"allocated":1165893632,"data_stored":1165233676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1634584114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,5,7,14,11,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22544384,"num_objects":-2752,"num_object_clones":0,"num_object_copies":-11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005051"},"pg_stats":[{"pgid":"1.f","version":"33'7806","reported_seq":7852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.943476+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:10.943476+0000","last_peered":"2024-09-15T21:09:10.943476+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:10.943476+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7806,"log_dups_size":0,"ondisk_log_size":7806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37240832,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7989","reported_seq":10558,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.297821+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:10.297821+0000","last_peered":"2024-09-15T21:09:10.297821+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:10.297821+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377292,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7861","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.929595+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:12.929595+0000","last_peered":"2024-09-15T21:09:12.929595+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:12.929595+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37444254,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7860,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7884","reported_seq":7953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.214403+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:10.214403+0000","last_peered":"2024-09-15T21:09:10.214403+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:10.214403+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37451020,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7882,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8011","reported_seq":8067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.896082+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:10.896082+0000","last_peered":"2024-09-15T21:09:10.896082+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:10.896082+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38073612,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7878","reported_seq":7923,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.930947+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:10.930947+0000","last_peered":"2024-09-15T21:09:10.930947+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:10.930947+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36847616,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8099","reported_seq":8169,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.272462+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:10.272462+0000","last_peered":"2024-09-15T21:09:10.272462+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:10.272462+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38278412,"num_objects":4673,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2307}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8127","reported_seq":8196,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.227576+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:10.227576+0000","last_peered":"2024-09-15T21:09:10.227576+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:10.227576+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958924,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7929","reported_seq":7989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.833155+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:10.833155+0000","last_peered":"2024-09-15T21:09:10.833155+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:10.833155+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37046808,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7988","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.073977+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:13.073977+0000","last_peered":"2024-09-15T21:09:13.073977+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:13.073977+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7828","reported_seq":12492,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:10.960445+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:10.960445+0000","last_peered":"2024-09-15T21:09:10.960445+0000","last_clean":"2024-09-15T21:09:10.960445+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:10.960445+0000","last_undegraded":"2024-09-15T21:09:10.960445+0000","last_fullsized":"2024-09-15T21:09:10.960445+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7828,"log_dups_size":0,"ondisk_log_size":7828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37126144,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7839","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.852910+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:10.852910+0000","last_peered":"2024-09-15T21:09:10.852910+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:10.852910+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7839,"log_dups_size":0,"ondisk_log_size":7839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37583494,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7908","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.800568+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:12.800568+0000","last_peered":"2024-09-15T21:09:12.800568+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:12.800568+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37624454,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7837","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.733180+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:10.733180+0000","last_peered":"2024-09-15T21:09:10.733180+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:10.733180+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7837,"log_dups_size":0,"ondisk_log_size":7837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639436,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7835,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7998","reported_seq":8081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.084540+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:13.084540+0000","last_peered":"2024-09-15T21:09:13.084540+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:13.084540+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7996,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7959","reported_seq":8027,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.314836+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:10.314836+0000","last_peered":"2024-09-15T21:09:10.314836+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:10.314836+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37598476,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":601118510,"num_objects":73384,"num_object_clones":0,"num_object_copies":293536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121028,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1162412032,"data_stored":1162412032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126941,"ondisk_log_size":126941,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719604,"kb_used_data":295004,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93652236,"statfs":{"total":96636764160,"available":95899889664,"internally_reserved":0,"allocated":302084096,"data_stored":301919107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720788,"kb_used_data":292548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93651052,"statfs":{"total":96636764160,"available":95898677248,"internally_reserved":0,"allocated":299569152,"data_stored":299404163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566184,"kb_used_data":255512,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93805656,"statfs":{"total":96636764160,"available":96056991744,"internally_reserved":0,"allocated":261644288,"data_stored":261479299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728356,"kb_used_data":295504,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93643484,"statfs":{"total":96636764160,"available":95890927616,"internally_reserved":0,"allocated":302596096,"data_stored":302431107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,5,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301547520,"data_stored":301547520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261361664,"data_stored":261361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298655744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300847104,"data_stored":300847104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:14.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:14.959 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:14.959 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:14.983 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-15T21:09:14.048357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":601118510,"num_objects":73384,"num_object_clones":0,"num_object_copies":293536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121028,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126941,"ondisk_log_size":126941,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2734932,"kb_used_data":1138568,"kb_used_omap":14,"kb_used_meta":1596273,"kb_avail":374752428,"statfs":{"total":386547056640,"available":383746486272,"internally_reserved":0,"allocated":1165893632,"data_stored":1165233676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1634584114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,5,7,14,11,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22544384,"num_objects":-2752,"num_object_clones":0,"num_object_copies":-11008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2752,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005051"},"pg_stats":[{"pgid":"1.f","version":"33'7806","reported_seq":7852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.943476+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:10.943476+0000","last_peered":"2024-09-15T21:09:10.943476+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:10.943476+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7806,"log_dups_size":0,"ondisk_log_size":7806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37240832,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'7989","reported_seq":10558,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.297821+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:10.297821+0000","last_peered":"2024-09-15T21:09:10.297821+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:10.297821+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7989,"log_dups_size":0,"ondisk_log_size":7989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37377292,"num_objects":4563,"num_object_clones":0,"num_object_copies":18252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1132,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7987,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3431}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7861","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.929595+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:12.929595+0000","last_peered":"2024-09-15T21:09:12.929595+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:12.929595+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37444254,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7860,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7884","reported_seq":7953,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.214403+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:10.214403+0000","last_peered":"2024-09-15T21:09:10.214403+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:10.214403+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7884,"log_dups_size":0,"ondisk_log_size":7884,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37451020,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7882,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2238}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8011","reported_seq":8067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.896082+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:10.896082+0000","last_peered":"2024-09-15T21:09:10.896082+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:10.896082+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38073612,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7878","reported_seq":7923,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.930947+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:10.930947+0000","last_peered":"2024-09-15T21:09:10.930947+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:10.930947+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36847616,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8099","reported_seq":8169,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.272462+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:10.272462+0000","last_peered":"2024-09-15T21:09:10.272462+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:10.272462+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38278412,"num_objects":4673,"num_object_clones":0,"num_object_copies":18692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2307}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8127","reported_seq":8196,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.227576+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:10.227576+0000","last_peered":"2024-09-15T21:09:10.227576+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:10.227576+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8127,"log_dups_size":0,"ondisk_log_size":8127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37958924,"num_objects":4634,"num_object_clones":0,"num_object_copies":18536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8125,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2160}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7929","reported_seq":7989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.833155+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:10.833155+0000","last_peered":"2024-09-15T21:09:10.833155+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:10.833155+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37046808,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7988","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.073977+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:13.073977+0000","last_peered":"2024-09-15T21:09:13.073977+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:13.073977+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7828","reported_seq":12492,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:10.960445+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:10.960445+0000","last_peered":"2024-09-15T21:09:10.960445+0000","last_clean":"2024-09-15T21:09:10.960445+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:10.960445+0000","last_undegraded":"2024-09-15T21:09:10.960445+0000","last_fullsized":"2024-09-15T21:09:10.960445+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7828,"log_dups_size":0,"ondisk_log_size":7828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37126144,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7839","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.852910+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:10.852910+0000","last_peered":"2024-09-15T21:09:10.852910+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:10.852910+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7839,"log_dups_size":0,"ondisk_log_size":7839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37583494,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7908","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.800568+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:12.800568+0000","last_peered":"2024-09-15T21:09:12.800568+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:12.800568+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37624454,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7837","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.733180+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:10.733180+0000","last_peered":"2024-09-15T21:09:10.733180+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:10.733180+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7837,"log_dups_size":0,"ondisk_log_size":7837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639436,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7835,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7998","reported_seq":8081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.084540+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:13.084540+0000","last_peered":"2024-09-15T21:09:13.084540+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:13.084540+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7996,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'7959","reported_seq":8027,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.314836+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:10.314836+0000","last_peered":"2024-09-15T21:09:10.314836+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:10.314836+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7959,"log_dups_size":0,"ondisk_log_size":7959,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37598476,"num_objects":4590,"num_object_clones":0,"num_object_copies":18360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2275}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":601118510,"num_objects":73384,"num_object_clones":0,"num_object_copies":293536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121028,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":10223616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1162412032,"data_stored":1162412032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126941,"ondisk_log_size":126941,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719604,"kb_used_data":295004,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93652236,"statfs":{"total":96636764160,"available":95899889664,"internally_reserved":0,"allocated":302084096,"data_stored":301919107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720788,"kb_used_data":292548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93651052,"statfs":{"total":96636764160,"available":95898677248,"internally_reserved":0,"allocated":299569152,"data_stored":299404163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566184,"kb_used_data":255512,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93805656,"statfs":{"total":96636764160,"available":96056991744,"internally_reserved":0,"allocated":261644288,"data_stored":261479299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247885,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728356,"kb_used_data":295504,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93643484,"statfs":{"total":96636764160,"available":95890927616,"internally_reserved":0,"allocated":302596096,"data_stored":302431107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":443215566},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,5,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":301547520,"data_stored":301547520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261361664,"data_stored":261361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298655744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300847104,"data_stored":300847104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7806", 'reported_seq': 7852, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.943476+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:10.943476+0000', 'last_peered': '2024-09-15T21:09:10.943476+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:10.943476+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7806, 'log_dups_size': 0, 'ondisk_log_size': 7806, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37240832, 'num_objects': 4546, 'num_object_clones': 0, 'num_object_copies': 18184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4546, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7806, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2270}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'7989", 'reported_seq': 10558, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.297821+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:10.297821+0000', 'last_peered': '2024-09-15T21:09:10.297821+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:10.297821+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7989, 'log_dups_size': 0, 'ondisk_log_size': 7989, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37377292, 'num_objects': 4563, 'num_object_clones': 0, 'num_object_copies': 18252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1132, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4563, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7987, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1252, 'num_bytes_recovered': 10223616, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3431}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:14.984 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7861", 'reported_seq': 7941, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:12.929595+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:12.929595+0000', 'last_peered': '2024-09-15T21:09:12.929595+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:12.929595+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7861, 'log_dups_size': 0, 'ondisk_log_size': 7861, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37444254, 'num_objects': 4572, 'num_object_clones': 0, 'num_object_copies': 18288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4572, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7860, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2314}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7884", 'reported_seq': 7953, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.214403+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:10.214403+0000', 'last_peered': '2024-09-15T21:09:10.214403+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:10.214403+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7884, 'log_dups_size': 0, 'ondisk_log_size': 7884, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37451020, 'num_objects': 4572, 'num_object_clones': 0, 'num_object_copies': 18288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7882, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2238}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8011", 'reported_seq': 8067, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.896082+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:10.896082+0000', 'last_peered': '2024-09-15T21:09:10.896082+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:10.896082+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8011, 'log_dups_size': 0, 'ondisk_log_size': 8011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38073612, 'num_objects': 4648, 'num_object_clones': 0, 'num_object_copies': 18592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4648, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8009, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2337}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7878", 'reported_seq': 7923, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.930947+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:10.930947+0000', 'last_peered': '2024-09-15T21:09:10.930947+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:10.930947+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7878, 'log_dups_size': 0, 'ondisk_log_size': 7878, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36847616, 'num_objects': 4498, 'num_object_clones': 0, 'num_object_copies': 17992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7878, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2182}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8099", 'reported_seq': 8169, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.272462+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:10.272462+0000', 'last_peered': '2024-09-15T21:09:10.272462+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:10.272462+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8099, 'log_dups_size': 0, 'ondisk_log_size': 8099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38278412, 'num_objects': 4673, 'num_object_clones': 0, 'num_object_copies': 18692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8097, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2307}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:14.985 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8127", 'reported_seq': 8196, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.227576+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:10.227576+0000', 'last_peered': '2024-09-15T21:09:10.227576+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:10.227576+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8127, 'log_dups_size': 0, 'ondisk_log_size': 8127, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37958924, 'num_objects': 4634, 'num_object_clones': 0, 'num_object_copies': 18536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4634, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8125, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2160}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'7929", 'reported_seq': 7989, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.833155+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:10.833155+0000', 'last_peered': '2024-09-15T21:09:10.833155+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:10.833155+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7929, 'log_dups_size': 0, 'ondisk_log_size': 7929, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37046808, 'num_objects': 4523, 'num_object_clones': 0, 'num_object_copies': 18092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7926, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'7988", 'reported_seq': 8069, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:13.073977+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:13.073977+0000', 'last_peered': '2024-09-15T21:09:13.073977+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:13.073977+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7988, 'log_dups_size': 0, 'ondisk_log_size': 7988, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37467404, 'num_objects': 4574, 'num_object_clones': 0, 'num_object_copies': 18296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4574, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7986, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2239}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:14.986 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7839", 'reported_seq': 7879, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.852910+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:10.852910+0000', 'last_peered': '2024-09-15T21:09:10.852910+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:10.852910+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7839, 'log_dups_size': 0, 'ondisk_log_size': 7839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37583494, 'num_objects': 4588, 'num_object_clones': 0, 'num_object_copies': 18352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4588, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7838, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7908", 'reported_seq': 7988, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:12.800568+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:12.800568+0000', 'last_peered': '2024-09-15T21:09:12.800568+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:12.800568+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7908, 'log_dups_size': 0, 'ondisk_log_size': 7908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37624454, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 18372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7907, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7837", 'reported_seq': 7894, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.733180+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:10.733180+0000', 'last_peered': '2024-09-15T21:09:10.733180+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:10.733180+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7837, 'log_dups_size': 0, 'ondisk_log_size': 7837, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37639436, 'num_objects': 4595, 'num_object_clones': 0, 'num_object_copies': 18380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7835, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2316}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'7998", 'reported_seq': 8081, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:13.084540+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:13.084540+0000', 'last_peered': '2024-09-15T21:09:13.084540+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:13.084540+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7998, 'log_dups_size': 0, 'ondisk_log_size': 7998, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38360332, 'num_objects': 4683, 'num_object_clones': 0, 'num_object_copies': 18732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4683, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7996, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2402}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:14.987 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'7959", 'reported_seq': 8027, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:10.314836+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:10.314836+0000', 'last_peered': '2024-09-15T21:09:10.314836+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:10.314836+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7959, 'log_dups_size': 0, 'ondisk_log_size': 7959, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37598476, 'num_objects': 4590, 'num_object_clones': 0, 'num_object_copies': 18360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4590, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7957, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2275}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:14.987 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:15.724 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:15.724 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33573/293536 objects degraded (11.437%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":421,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":73384,"data_bytes":601118510,"bytes_used":2800570368,"bytes_avail":383746486272,"bytes_total":386547056640,"degraded_objects":33573,"degraded_total":293536,"degraded_ratio":0.11437438678731059,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":65508,"recovering_keys_per_sec":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":229},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:15.752 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:15.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:16.434 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:16.434 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:16.454 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-09-15T21:09:16.049044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597759790,"num_objects":72974,"num_object_clones":0,"num_object_copies":291896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121438,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127351,"ondisk_log_size":127351,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2734200,"kb_used_data":1133680,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374753160,"statfs":{"total":386547056640,"available":383747235840,"internally_reserved":0,"allocated":1160888320,"data_stored":1160228364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,10,10,9,13,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23486464,"num_objects":-2867,"num_object_clones":0,"num_object_copies":-11468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2867,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004991"},"pg_stats":[{"pgid":"1.f","version":"33'7806","reported_seq":7852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.943476+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:10.943476+0000","last_peered":"2024-09-15T21:09:10.943476+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:10.943476+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7806,"log_dups_size":0,"ondisk_log_size":7806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37240832,"num_objects":4546,"num_object_clones":0,"num_object_copies":18184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7806,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2270}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8066","reported_seq":10735,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.403960+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:15.403960+0000","last_peered":"2024-09-15T21:09:15.403960+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:15.403960+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36746508,"num_objects":4486,"num_object_clones":0,"num_object_copies":17944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1302,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7861","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.929595+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:12.929595+0000","last_peered":"2024-09-15T21:09:12.929595+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:12.929595+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37444254,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7860,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7975","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.325262+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:15.325262+0000","last_peered":"2024-09-15T21:09:15.325262+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:15.325262+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36705548,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8011","reported_seq":8067,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.896082+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:10.896082+0000","last_peered":"2024-09-15T21:09:10.896082+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:10.896082+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38073612,"num_objects":4648,"num_object_clones":0,"num_object_copies":18592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2337}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7878","reported_seq":7923,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.930947+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:10.930947+0000","last_peered":"2024-09-15T21:09:10.930947+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:10.930947+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7878,"log_dups_size":0,"ondisk_log_size":7878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36847616,"num_objects":4498,"num_object_clones":0,"num_object_copies":17992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7878,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8174","reported_seq":8244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.293573+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:15.293573+0000","last_peered":"2024-09-15T21:09:15.293573+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:15.293573+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37664012,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8213","reported_seq":8282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.332975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:15.332975+0000","last_peered":"2024-09-15T21:09:15.332975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:15.332975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37254412,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8211,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'7929","reported_seq":7989,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.833155+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:10.833155+0000","last_peered":"2024-09-15T21:09:10.833155+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:10.833155+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37046808,"num_objects":4523,"num_object_clones":0,"num_object_copies":18092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7988","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.073977+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:13.073977+0000","last_peered":"2024-09-15T21:09:13.073977+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:13.073977+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7828","reported_seq":12492,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:10.960445+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:10.960445+0000","last_peered":"2024-09-15T21:09:10.960445+0000","last_clean":"2024-09-15T21:09:10.960445+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:10.960445+0000","last_undegraded":"2024-09-15T21:09:10.960445+0000","last_fullsized":"2024-09-15T21:09:10.960445+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7828,"log_dups_size":0,"ondisk_log_size":7828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37126144,"num_objects":4532,"num_object_clones":0,"num_object_copies":18128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7839","reported_seq":7879,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.852910+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:10.852910+0000","last_peered":"2024-09-15T21:09:10.852910+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:10.852910+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7839,"log_dups_size":0,"ondisk_log_size":7839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37583494,"num_objects":4588,"num_object_clones":0,"num_object_copies":18352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7838,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2332}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7908","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.800568+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:12.800568+0000","last_peered":"2024-09-15T21:09:12.800568+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:12.800568+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37624454,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7837","reported_seq":7894,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:10.733180+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:10.733180+0000","last_peered":"2024-09-15T21:09:10.733180+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:10.733180+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7837,"log_dups_size":0,"ondisk_log_size":7837,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37639436,"num_objects":4595,"num_object_clones":0,"num_object_copies":18380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7835,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2316}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7998","reported_seq":8081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.084540+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:13.084540+0000","last_peered":"2024-09-15T21:09:13.084540+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:13.084540+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7996,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8040","reported_seq":8109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.142073+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:15.142073+0000","last_peered":"2024-09-15T21:09:15.142073+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:15.142073+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":597759790,"num_objects":72974,"num_object_clones":0,"num_object_copies":291896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72974,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121438,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1157361664,"data_stored":1157361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127351,"ondisk_log_size":127351,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719604,"kb_used_data":295004,"kb_used_omap":4,"kb_used_meta":424571,"kb_avail":93652236,"statfs":{"total":96636764160,"available":95899889664,"internally_reserved":0,"allocated":302084096,"data_stored":301919107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":434761424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720788,"kb_used_data":292548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93651052,"statfs":{"total":96636764160,"available":95898677248,"internally_reserved":0,"allocated":299569152,"data_stored":299404163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566184,"kb_used_data":255512,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93805656,"statfs":{"total":96636764160,"available":96056991744,"internally_reserved":0,"allocated":261644288,"data_stored":261479299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727624,"kb_used_data":290616,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93644216,"statfs":{"total":96636764160,"available":95891677184,"internally_reserved":0,"allocated":297590784,"data_stored":297425795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,5,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296497152,"data_stored":296497152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261361664,"data_stored":261361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298655744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":300847104,"data_stored":300847104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:19.455 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:20.052 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:20.052 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:20.073 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-15T21:09:18.049656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593696558,"num_objects":72478,"num_object_clones":0,"num_object_copies":289912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121934,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127847,"ondisk_log_size":127847,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732688,"kb_used_data":1127996,"kb_used_omap":14,"kb_used_meta":1604593,"kb_avail":374754672,"statfs":{"total":386547056640,"available":383748784128,"internally_reserved":0,"allocated":1155067904,"data_stored":1154407948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1643103794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,7,10,10,12,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19963904,"num_objects":-2437,"num_object_clones":0,"num_object_copies":-9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004721"},"pg_stats":[{"pgid":"1.f","version":"33'7874","reported_seq":7920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:16.047711+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:16.047711+0000","last_peered":"2024-09-15T21:09:16.047711+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:16.047711+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36683776,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8066","reported_seq":10735,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.403960+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:15.403960+0000","last_peered":"2024-09-15T21:09:15.403960+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:15.403960+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36746508,"num_objects":4486,"num_object_clones":0,"num_object_copies":17944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1302,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7861","reported_seq":7941,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.929595+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:12.929595+0000","last_peered":"2024-09-15T21:09:12.929595+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:12.929595+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7861,"log_dups_size":0,"ondisk_log_size":7861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37444254,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7860,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2314}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7975","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.325262+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:15.325262+0000","last_peered":"2024-09-15T21:09:15.325262+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:15.325262+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36705548,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8087","reported_seq":8143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:16.050753+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:16.050753+0000","last_peered":"2024-09-15T21:09:16.050753+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:16.050753+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37451020,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7943","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.585729+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:15.585729+0000","last_peered":"2024-09-15T21:09:15.585729+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:15.585729+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8174","reported_seq":8244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.293573+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:15.293573+0000","last_peered":"2024-09-15T21:09:15.293573+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:15.293573+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37664012,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8213","reported_seq":8282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.332975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:15.332975+0000","last_peered":"2024-09-15T21:09:15.332975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:15.332975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37254412,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8211,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8001","reported_seq":8061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.895029+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:15.895029+0000","last_peered":"2024-09-15T21:09:15.895029+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:15.895029+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8001,"log_dups_size":0,"ondisk_log_size":8001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36456984,"num_objects":4451,"num_object_clones":0,"num_object_copies":17804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'7988","reported_seq":8069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.073977+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:13.073977+0000","last_peered":"2024-09-15T21:09:13.073977+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:13.073977+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37467404,"num_objects":4574,"num_object_clones":0,"num_object_copies":18296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7986,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2239}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7898","reported_seq":12562,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:15.804319+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:15.804319+0000","last_peered":"2024-09-15T21:09:15.804319+0000","last_clean":"2024-09-15T21:09:15.804319+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:15.804319+0000","last_undegraded":"2024-09-15T21:09:15.804319+0000","last_fullsized":"2024-09-15T21:09:15.804319+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36552704,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2009,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7908","reported_seq":7948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.976705+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:15.976705+0000","last_peered":"2024-09-15T21:09:15.976705+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:15.976705+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37018246,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7908","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:12.800568+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:12.800568+0000","last_peered":"2024-09-15T21:09:12.800568+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:12.800568+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37624454,"num_objects":4593,"num_object_clones":0,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7913","reported_seq":7970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.976813+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:15.976813+0000","last_peered":"2024-09-15T21:09:15.976813+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:15.976813+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016844,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2240}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'7998","reported_seq":8081,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:13.084540+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:13.084540+0000","last_peered":"2024-09-15T21:09:13.084540+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:13.084540+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7998,"log_dups_size":0,"ondisk_log_size":7998,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38360332,"num_objects":4683,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7996,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2402}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8040","reported_seq":8109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.142073+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:15.142073+0000","last_peered":"2024-09-15T21:09:15.142073+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:15.142073+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":593696558,"num_objects":72478,"num_object_clones":0,"num_object_copies":289912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121934,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1152409600,"data_stored":1152409600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127847,"ondisk_log_size":127847,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718092,"kb_used_data":289320,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93653748,"statfs":{"total":96636764160,"available":95901437952,"internally_reserved":0,"allocated":296263680,"data_stored":296098691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720788,"kb_used_data":292548,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":93651052,"statfs":{"total":96636764160,"available":95898677248,"internally_reserved":0,"allocated":299569152,"data_stored":299404163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051648,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566184,"kb_used_data":255512,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93805656,"statfs":{"total":96636764160,"available":96056991744,"internally_reserved":0,"allocated":261644288,"data_stored":261479299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727624,"kb_used_data":290616,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93644216,"statfs":{"total":96636764160,"available":95891677184,"internally_reserved":0,"allocated":297590784,"data_stored":297425795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,5,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296497152,"data_stored":296497152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261361664,"data_stored":261361664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":298655744,"data_stored":298655744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":295895040,"data_stored":295895040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:20.074 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:20.673 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:20.674 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:20.699 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-15T21:09:20.050362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591247150,"num_objects":72179,"num_object_clones":0,"num_object_copies":288716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122233,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128146,"ondisk_log_size":128146,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2723524,"kb_used_data":1122988,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374763836,"statfs":{"total":386547056640,"available":383758168064,"internally_reserved":0,"allocated":1149939712,"data_stored":1149279756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,8,8,14,12,11,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22413312,"num_objects":-2736,"num_object_clones":0,"num_object_copies":-10944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2736,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004608"},"pg_stats":[{"pgid":"1.f","version":"33'7874","reported_seq":7920,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:16.047711+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:16.047711+0000","last_peered":"2024-09-15T21:09:16.047711+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:16.047711+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7874,"log_dups_size":0,"ondisk_log_size":7874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36683776,"num_objects":4478,"num_object_clones":0,"num_object_copies":17912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7874,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8066","reported_seq":10735,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.403960+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:15.403960+0000","last_peered":"2024-09-15T21:09:15.403960+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:15.403960+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8066,"log_dups_size":0,"ondisk_log_size":8066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36746508,"num_objects":4486,"num_object_clones":0,"num_object_copies":17944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1082,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8064,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1302,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7944","reported_seq":8023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:17.996169+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:17.996169+0000","last_peered":"2024-09-15T21:09:17.996169+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:17.996169+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36764318,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7943,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'7975","reported_seq":8044,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.325262+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:15.325262+0000","last_peered":"2024-09-15T21:09:15.325262+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:15.325262+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7975,"log_dups_size":0,"ondisk_log_size":7975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36705548,"num_objects":4481,"num_object_clones":0,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7973,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8087","reported_seq":8143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:16.050753+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:16.050753+0000","last_peered":"2024-09-15T21:09:16.050753+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:16.050753+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8087,"log_dups_size":0,"ondisk_log_size":8087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37451020,"num_objects":4572,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8085,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2261}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'7943","reported_seq":7988,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.585729+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:15.585729+0000","last_peered":"2024-09-15T21:09:15.585729+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:15.585729+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7943,"log_dups_size":0,"ondisk_log_size":7943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36315136,"num_objects":4433,"num_object_clones":0,"num_object_copies":17732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7943,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8174","reported_seq":8244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.293573+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:15.293573+0000","last_peered":"2024-09-15T21:09:15.293573+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:15.293573+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8174,"log_dups_size":0,"ondisk_log_size":8174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37664012,"num_objects":4598,"num_object_clones":0,"num_object_copies":18392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8172,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2232}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8213","reported_seq":8282,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.332975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:15.332975+0000","last_peered":"2024-09-15T21:09:15.332975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:15.332975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8213,"log_dups_size":0,"ondisk_log_size":8213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37254412,"num_objects":4548,"num_object_clones":0,"num_object_copies":18192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8211,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2074}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8001","reported_seq":8061,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.895029+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:15.895029+0000","last_peered":"2024-09-15T21:09:15.895029+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:15.895029+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8001,"log_dups_size":0,"ondisk_log_size":8001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36456984,"num_objects":4451,"num_object_clones":0,"num_object_copies":17804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2119}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8061","reported_seq":8143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:17.900213+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:17.900213+0000","last_peered":"2024-09-15T21:09:17.900213+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:17.900213+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36869388,"num_objects":4501,"num_object_clones":0,"num_object_copies":18004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7898","reported_seq":12562,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:15.804319+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:15.804319+0000","last_peered":"2024-09-15T21:09:15.804319+0000","last_clean":"2024-09-15T21:09:15.804319+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:15.804319+0000","last_undegraded":"2024-09-15T21:09:15.804319+0000","last_fullsized":"2024-09-15T21:09:15.804319+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7898,"log_dups_size":0,"ondisk_log_size":7898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36552704,"num_objects":4462,"num_object_clones":0,"num_object_copies":17848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2009,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7908","reported_seq":7948,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.976705+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:15.976705+0000","last_peered":"2024-09-15T21:09:15.976705+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:15.976705+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7908,"log_dups_size":0,"ondisk_log_size":7908,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37018246,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7907,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2263}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7982","reported_seq":8062,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:18.042034+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:18.042034+0000","last_peered":"2024-09-15T21:09:18.042034+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:18.042034+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7982,"log_dups_size":0,"ondisk_log_size":7982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37018246,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7981,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'7913","reported_seq":7970,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.976813+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:15.976813+0000","last_peered":"2024-09-15T21:09:15.976813+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:15.976813+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7913,"log_dups_size":0,"ondisk_log_size":7913,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37016844,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2240}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8067","reported_seq":8150,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:18.017283+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:18.017283+0000","last_peered":"2024-09-15T21:09:18.017283+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:18.017283+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37795084,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8065,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8040","reported_seq":8109,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:15.142073+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:15.142073+0000","last_peered":"2024-09-15T21:09:15.142073+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:15.142073+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8040,"log_dups_size":0,"ondisk_log_size":8040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36934924,"num_objects":4509,"num_object_clones":0,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":591247150,"num_objects":72179,"num_object_clones":0,"num_object_copies":288716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72179,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122233,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1324,"num_bytes_recovered":10633216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1147195392,"data_stored":1147195392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128146,"ondisk_log_size":128146,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718092,"kb_used_data":289320,"kb_used_omap":4,"kb_used_meta":428731,"kb_avail":93653748,"statfs":{"total":96636764160,"available":95901437952,"internally_reserved":0,"allocated":296263680,"data_stored":296098691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":439021264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,5,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720220,"kb_used_data":287816,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93651620,"statfs":{"total":96636764160,"available":95899258880,"internally_reserved":0,"allocated":294723584,"data_stored":294558595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,2,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557588,"kb_used_data":255236,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93814252,"statfs":{"total":96636764160,"available":96065794048,"internally_reserved":0,"allocated":261361664,"data_stored":261196675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727624,"kb_used_data":290616,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93644216,"statfs":{"total":96636764160,"available":95891677184,"internally_reserved":0,"allocated":297590784,"data_stored":297425795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":447475406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,7,5,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":296497152,"data_stored":296497152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261042176,"data_stored":261042176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293761024,"data_stored":293761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":295895040,"data_stored":295895040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7874", 'reported_seq': 7920, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:16.047711+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:16.047711+0000', 'last_peered': '2024-09-15T21:09:16.047711+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:16.047711+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7874, 'log_dups_size': 0, 'ondisk_log_size': 7874, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36683776, 'num_objects': 4478, 'num_object_clones': 0, 'num_object_copies': 17912, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4478, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7874, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2202}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8066", 'reported_seq': 10735, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.403960+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:15.403960+0000', 'last_peered': '2024-09-15T21:09:15.403960+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:15.403960+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8066, 'log_dups_size': 0, 'ondisk_log_size': 8066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36746508, 'num_objects': 4486, 'num_object_clones': 0, 'num_object_copies': 17944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1082, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8064, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1302, 'num_bytes_recovered': 10633216, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3404}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'7944", 'reported_seq': 8023, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:17.996169+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:17.996169+0000', 'last_peered': '2024-09-15T21:09:17.996169+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:17.996169+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7944, 'log_dups_size': 0, 'ondisk_log_size': 7944, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36764318, 'num_objects': 4489, 'num_object_clones': 0, 'num_object_copies': 17956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4489, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 7943, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2231}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:20.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'7975", 'reported_seq': 8044, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.325262+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:15.325262+0000', 'last_peered': '2024-09-15T21:09:15.325262+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:15.325262+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7975, 'log_dups_size': 0, 'ondisk_log_size': 7975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36705548, 'num_objects': 4481, 'num_object_clones': 0, 'num_object_copies': 17924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4481, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7973, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8087", 'reported_seq': 8143, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:16.050753+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:16.050753+0000', 'last_peered': '2024-09-15T21:09:16.050753+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:16.050753+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8087, 'log_dups_size': 0, 'ondisk_log_size': 8087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37451020, 'num_objects': 4572, 'num_object_clones': 0, 'num_object_copies': 18288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4572, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8085, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2261}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'7943", 'reported_seq': 7988, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.585729+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:15.585729+0000', 'last_peered': '2024-09-15T21:09:15.585729+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:15.585729+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7943, 'log_dups_size': 0, 'ondisk_log_size': 7943, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36315136, 'num_objects': 4433, 'num_object_clones': 0, 'num_object_copies': 17732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7943, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2117}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8174", 'reported_seq': 8244, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.293573+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:15.293573+0000', 'last_peered': '2024-09-15T21:09:15.293573+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:15.293573+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8174, 'log_dups_size': 0, 'ondisk_log_size': 8174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37664012, 'num_objects': 4598, 'num_object_clones': 0, 'num_object_copies': 18392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4598, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8172, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2232}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8213", 'reported_seq': 8282, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.332975+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:15.332975+0000', 'last_peered': '2024-09-15T21:09:15.332975+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:15.332975+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8213, 'log_dups_size': 0, 'ondisk_log_size': 8213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37254412, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 18192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8211, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8001", 'reported_seq': 8061, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.895029+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:15.895029+0000', 'last_peered': '2024-09-15T21:09:15.895029+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:15.895029+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8001, 'log_dups_size': 0, 'ondisk_log_size': 8001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36456984, 'num_objects': 4451, 'num_object_clones': 0, 'num_object_copies': 17804, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4451, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7998, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2119}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8061", 'reported_seq': 8143, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:17.900213+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:17.900213+0000', 'last_peered': '2024-09-15T21:09:17.900213+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:17.900213+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8061, 'log_dups_size': 0, 'ondisk_log_size': 8061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36869388, 'num_objects': 4501, 'num_object_clones': 0, 'num_object_copies': 18004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4501, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8059, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2166}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7908", 'reported_seq': 7948, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.976705+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:15.976705+0000', 'last_peered': '2024-09-15T21:09:15.976705+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:15.976705+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7908, 'log_dups_size': 0, 'ondisk_log_size': 7908, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37018246, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7907, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2263}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'7982", 'reported_seq': 8062, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:18.042034+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:18.042034+0000', 'last_peered': '2024-09-15T21:09:18.042034+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:18.042034+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7982, 'log_dups_size': 0, 'ondisk_log_size': 7982, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37018246, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7981, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2233}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'7913", 'reported_seq': 7970, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.976813+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:15.976813+0000', 'last_peered': '2024-09-15T21:09:15.976813+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:15.976813+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7913, 'log_dups_size': 0, 'ondisk_log_size': 7913, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37016844, 'num_objects': 4519, 'num_object_clones': 0, 'num_object_copies': 18076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4519, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7911, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8067", 'reported_seq': 8150, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:18.017283+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:18.017283+0000', 'last_peered': '2024-09-15T21:09:18.017283+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:18.017283+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8067, 'log_dups_size': 0, 'ondisk_log_size': 8067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37795084, 'num_objects': 4614, 'num_object_clones': 0, 'num_object_copies': 18456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8065, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2333}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:20.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8040", 'reported_seq': 8109, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:15.142073+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:15.142073+0000', 'last_peered': '2024-09-15T21:09:15.142073+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:15.142073+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8040, 'log_dups_size': 0, 'ondisk_log_size': 8040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36934924, 'num_objects': 4509, 'num_object_clones': 0, 'num_object_copies': 18036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4509, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8038, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:20.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:21.447 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:21.447 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33523/288716 objects degraded (11.611%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":427,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":72179,"data_bytes":591247150,"bytes_used":2788888576,"bytes_avail":383758168064,"bytes_total":386547056640,"degraded_objects":33523,"degraded_total":288716,"degraded_ratio":0.11611064159935715,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":66875,"recovering_keys_per_sec":0,"num_objects_recovered":98,"num_bytes_recovered":802816,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":227},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:21.479 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:21.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:22.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:22.070 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:22.088 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-15T21:09:22.050934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":583415598,"num_objects":71223,"num_object_clones":0,"num_object_copies":284892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123189,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129102,"ondisk_log_size":129102,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704880,"kb_used_data":1112676,"kb_used_omap":14,"kb_used_meta":1592113,"kb_avail":374782480,"statfs":{"total":386547056640,"available":383777259520,"internally_reserved":0,"allocated":1139380224,"data_stored":1138720268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1630324274},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,5,1,15,10,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27820032,"num_objects":-3396,"num_object_clones":0,"num_object_copies":-13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004376"},"pg_stats":[{"pgid":"1.f","version":"33'7954","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.193624+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:21.193624+0000","last_peered":"2024-09-15T21:09:21.193624+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:21.193624+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8162","reported_seq":10914,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.442950+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:20.442950+0000","last_peered":"2024-09-15T21:09:20.442950+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:20.442950+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8162,"log_dups_size":0,"ondisk_log_size":8162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960076,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1344,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'7944","reported_seq":8023,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:17.996169+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:17.996169+0000","last_peered":"2024-09-15T21:09:17.996169+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:17.996169+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7944,"log_dups_size":0,"ondisk_log_size":7944,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36764318,"num_objects":4489,"num_object_clones":0,"num_object_copies":17956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7943,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2231}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8058","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.280351+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:20.280351+0000","last_peered":"2024-09-15T21:09:20.280351+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:20.280351+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8058,"log_dups_size":0,"ondisk_log_size":8058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8156","reported_seq":8210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.037299+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:21.037299+0000","last_peered":"2024-09-15T21:09:21.037299+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:21.037299+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8008","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.164960+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:21.164960+0000","last_peered":"2024-09-15T21:09:21.164960+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:21.164960+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8242","reported_seq":8313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.428993+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:20.428993+0000","last_peered":"2024-09-15T21:09:20.428993+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:20.428993+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37106956,"num_objects":4530,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2164}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8296","reported_seq":8365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.389205+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:20.389205+0000","last_peered":"2024-09-15T21:09:20.389205+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:20.389205+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8076","reported_seq":8136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.151179+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:21.151179+0000","last_peered":"2024-09-15T21:09:21.151179+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:21.151179+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35842584,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8061","reported_seq":8143,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:17.900213+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:17.900213+0000","last_peered":"2024-09-15T21:09:17.900213+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:17.900213+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8061,"log_dups_size":0,"ondisk_log_size":8061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36869388,"num_objects":4501,"num_object_clones":0,"num_object_copies":18004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8059,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2166}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7988","reported_seq":12651,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:21.172568+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:21.172568+0000","last_peered":"2024-09-15T21:09:21.172568+0000","last_clean":"2024-09-15T21:09:21.172568+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:21.172568+0000","last_undegraded":"2024-09-15T21:09:21.172568+0000","last_fullsized":"2024-09-15T21:09:21.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7993","reported_seq":8033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.953686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:20.953686+0000","last_peered":"2024-09-15T21:09:20.953686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:20.953686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36321926,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'7982","reported_seq":8062,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:18.042034+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:18.042034+0000","last_peered":"2024-09-15T21:09:18.042034+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:18.042034+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7982,"log_dups_size":0,"ondisk_log_size":7982,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37018246,"num_objects":4519,"num_object_clones":0,"num_object_copies":18076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7981,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2233}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8000","reported_seq":8057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.988136+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:20.988136+0000","last_peered":"2024-09-15T21:09:20.988136+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:20.988136+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36304140,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8067","reported_seq":8150,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:18.017283+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:18.017283+0000","last_peered":"2024-09-15T21:09:18.017283+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:18.017283+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8067,"log_dups_size":0,"ondisk_log_size":8067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37795084,"num_objects":4614,"num_object_clones":0,"num_object_copies":18456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8065,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2333}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8115","reported_seq":8184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.351540+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:20.351540+0000","last_peered":"2024-09-15T21:09:20.351540+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:20.351540+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320524,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":583415598,"num_objects":71223,"num_object_clones":0,"num_object_copies":284892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123189,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1137192960,"data_stored":1137192960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129102,"ondisk_log_size":129102,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717444,"kb_used_data":284532,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93654396,"statfs":{"total":96636764160,"available":95902101504,"internally_reserved":0,"allocated":291360768,"data_stored":291195779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720220,"kb_used_data":287816,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93651620,"statfs":{"total":96636764160,"available":95899258880,"internally_reserved":0,"allocated":294723584,"data_stored":294558595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,1,2,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051649,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557588,"kb_used_data":255236,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93814252,"statfs":{"total":96636764160,"available":96065794048,"internally_reserved":0,"allocated":261361664,"data_stored":261196675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,3,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709628,"kb_used_data":285092,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93662212,"statfs":{"total":96636764160,"available":95910105088,"internally_reserved":0,"allocated":291934208,"data_stored":291769219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":434695886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,6,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291532800,"data_stored":291532800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":261042176,"data_stored":261042176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":293761024,"data_stored":293761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290856960,"data_stored":290856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:25.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:25.698 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:25.699 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:25.725 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-09-15T21:09:24.051468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581277486,"num_objects":70962,"num_object_clones":0,"num_object_copies":283848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123450,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129363,"ondisk_log_size":129363,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708276,"kb_used_data":1107756,"kb_used_omap":14,"kb_used_meta":1600433,"kb_avail":374779084,"statfs":{"total":386547056640,"available":383773782016,"internally_reserved":0,"allocated":1134342144,"data_stored":1133682188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1638843954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,2,16,6,13,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22396928,"num_objects":-2734,"num_object_clones":0,"num_object_copies":-10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2734,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":753664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004017"},"pg_stats":[{"pgid":"1.f","version":"33'7954","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.193624+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:21.193624+0000","last_peered":"2024-09-15T21:09:21.193624+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:21.193624+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8162","reported_seq":10914,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.442950+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:20.442950+0000","last_peered":"2024-09-15T21:09:20.442950+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:20.442950+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8162,"log_dups_size":0,"ondisk_log_size":8162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35960076,"num_objects":4390,"num_object_clones":0,"num_object_copies":17560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1040,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1344,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8018","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.186361+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:23.186361+0000","last_peered":"2024-09-15T21:09:23.186361+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:23.186361+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36158110,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8017,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2157}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8058","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.280351+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:20.280351+0000","last_peered":"2024-09-15T21:09:20.280351+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:20.280351+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8058,"log_dups_size":0,"ondisk_log_size":8058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36025612,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8056,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2064}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8156","reported_seq":8210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.037299+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:21.037299+0000","last_peered":"2024-09-15T21:09:21.037299+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:21.037299+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8008","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.164960+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:21.164960+0000","last_peered":"2024-09-15T21:09:21.164960+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:21.164960+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8242","reported_seq":8313,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.428993+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:20.428993+0000","last_peered":"2024-09-15T21:09:20.428993+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:20.428993+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8242,"log_dups_size":0,"ondisk_log_size":8242,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37106956,"num_objects":4530,"num_object_clones":0,"num_object_copies":18120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2164}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8296","reported_seq":8365,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.389205+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:20.389205+0000","last_peered":"2024-09-15T21:09:20.389205+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:20.389205+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8296,"log_dups_size":0,"ondisk_log_size":8296,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36574476,"num_objects":4465,"num_object_clones":0,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8294,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1991}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8076","reported_seq":8136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.151179+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:21.151179+0000","last_peered":"2024-09-15T21:09:21.151179+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:21.151179+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35842584,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8131","reported_seq":8213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.058570+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:23.058570+0000","last_peered":"2024-09-15T21:09:23.058570+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:23.058570+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8131,"log_dups_size":0,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295948,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7988","reported_seq":12651,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:21.172568+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:21.172568+0000","last_peered":"2024-09-15T21:09:21.172568+0000","last_clean":"2024-09-15T21:09:21.172568+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:21.172568+0000","last_undegraded":"2024-09-15T21:09:21.172568+0000","last_fullsized":"2024-09-15T21:09:21.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7993","reported_seq":8033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.953686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:20.953686+0000","last_peered":"2024-09-15T21:09:20.953686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:20.953686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36321926,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8047","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.210115+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:23.210115+0000","last_peered":"2024-09-15T21:09:23.210115+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:23.210115+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36485766,"num_objects":4454,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8000","reported_seq":8057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.988136+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:20.988136+0000","last_peered":"2024-09-15T21:09:20.988136+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:20.988136+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36304140,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8119","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.109036+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:23.109036+0000","last_peered":"2024-09-15T21:09:23.109036+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:23.109036+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37369100,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8115","reported_seq":8184,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.351540+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:20.351540+0000","last_peered":"2024-09-15T21:09:20.351540+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:20.351540+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8115,"log_dups_size":0,"ondisk_log_size":8115,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36320524,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8113,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2119}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":581277486,"num_objects":70962,"num_object_clones":0,"num_object_copies":283848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123450,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":10977280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1132081152,"data_stored":1132081152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129363,"ondisk_log_size":129363,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717444,"kb_used_data":284532,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93654396,"statfs":{"total":96636764160,"available":95902101504,"internally_reserved":0,"allocated":291360768,"data_stored":291195779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719696,"kb_used_data":283136,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93652144,"statfs":{"total":96636764160,"available":95899795456,"internally_reserved":0,"allocated":289931264,"data_stored":289766275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561508,"kb_used_data":254996,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93810332,"statfs":{"total":96636764160,"available":96061779968,"internally_reserved":0,"allocated":261115904,"data_stored":260950915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247887,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709628,"kb_used_data":285092,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93662212,"statfs":{"total":96636764160,"available":95910105088,"internally_reserved":0,"allocated":291934208,"data_stored":291769219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":434695886},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,6,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":291532800,"data_stored":291532800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260808704,"data_stored":260808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288882688,"data_stored":288882688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290856960,"data_stored":290856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:25.726 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:26.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:26.334 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:26.361 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-15T21:09:26.052097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578008878,"num_objects":70563,"num_object_clones":0,"num_object_copies":282252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123849,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129762,"ondisk_log_size":129762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725652,"kb_used_data":1102928,"kb_used_omap":14,"kb_used_meta":1622641,"kb_avail":374761708,"statfs":{"total":386547056640,"available":383755988992,"internally_reserved":0,"allocated":1129398272,"data_stored":1128738316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1661584946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,17,8,14,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23109632,"num_objects":-2821,"num_object_clones":0,"num_object_copies":-11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"1.f","version":"33'7954","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.193624+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:21.193624+0000","last_peered":"2024-09-15T21:09:21.193624+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:21.193624+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8236","reported_seq":11097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.494769+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:25.494769+0000","last_peered":"2024-09-15T21:09:25.494769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:25.494769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35353868,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8018","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.186361+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:23.186361+0000","last_peered":"2024-09-15T21:09:23.186361+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:23.186361+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36158110,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8017,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2157}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8141","reported_seq":8211,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.478136+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:25.478136+0000","last_peered":"2024-09-15T21:09:25.478136+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:25.478136+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8156","reported_seq":8210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.037299+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:21.037299+0000","last_peered":"2024-09-15T21:09:21.037299+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:21.037299+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8008","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.164960+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:21.164960+0000","last_peered":"2024-09-15T21:09:21.164960+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:21.164960+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8328","reported_seq":8399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.514233+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:25.514233+0000","last_peered":"2024-09-15T21:09:25.514233+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:25.514233+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402444,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8387","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.455088+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:25.455088+0000","last_peered":"2024-09-15T21:09:25.455088+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:25.455088+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35829004,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8385,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8076","reported_seq":8136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.151179+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:21.151179+0000","last_peered":"2024-09-15T21:09:21.151179+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:21.151179+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35842584,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8131","reported_seq":8213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.058570+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:23.058570+0000","last_peered":"2024-09-15T21:09:23.058570+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:23.058570+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8131,"log_dups_size":0,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295948,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7988","reported_seq":12651,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:21.172568+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:21.172568+0000","last_peered":"2024-09-15T21:09:21.172568+0000","last_clean":"2024-09-15T21:09:21.172568+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:21.172568+0000","last_undegraded":"2024-09-15T21:09:21.172568+0000","last_fullsized":"2024-09-15T21:09:21.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7993","reported_seq":8033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.953686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:20.953686+0000","last_peered":"2024-09-15T21:09:20.953686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:20.953686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36321926,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8047","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.210115+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:23.210115+0000","last_peered":"2024-09-15T21:09:23.210115+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:23.210115+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36485766,"num_objects":4454,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8000","reported_seq":8057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.988136+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:20.988136+0000","last_peered":"2024-09-15T21:09:20.988136+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:20.988136+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36304140,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8119","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.109036+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:23.109036+0000","last_peered":"2024-09-15T21:09:23.109036+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:23.109036+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37369100,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8180","reported_seq":8249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.403484+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:25.403484+0000","last_peered":"2024-09-15T21:09:25.403484+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:25.403484+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":578008878,"num_objects":70563,"num_object_clones":0,"num_object_copies":282252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123849,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1127116800,"data_stored":1127116800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129762,"ondisk_log_size":129762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717444,"kb_used_data":284532,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93654396,"statfs":{"total":96636764160,"available":95902101504,"internally_reserved":0,"allocated":291360768,"data_stored":291195779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719696,"kb_used_data":283136,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93652144,"statfs":{"total":96636764160,"available":95899795456,"internally_reserved":0,"allocated":289931264,"data_stored":289766275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561508,"kb_used_data":254996,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93810332,"statfs":{"total":96636764160,"available":96061779968,"internally_reserved":0,"allocated":261115904,"data_stored":260950915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727004,"kb_used_data":280264,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93644836,"statfs":{"total":96636764160,"available":95892312064,"internally_reserved":0,"allocated":286990336,"data_stored":286825347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":457436878},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286568448,"data_stored":286568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260808704,"data_stored":260808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288882688,"data_stored":288882688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290856960,"data_stored":290856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'7954", 'reported_seq': 7999, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:21.193624+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:21.193624+0000', 'last_peered': '2024-09-15T21:09:21.193624+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:21.193624+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7954, 'log_dups_size': 0, 'ondisk_log_size': 7954, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36028416, 'num_objects': 4398, 'num_object_clones': 0, 'num_object_copies': 17592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4398, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7954, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2122}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8236", 'reported_seq': 11097, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.494769+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:25.494769+0000', 'last_peered': '2024-09-15T21:09:25.494769+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:25.494769+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8236, 'log_dups_size': 0, 'ondisk_log_size': 8236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35353868, 'num_objects': 4316, 'num_object_clones': 0, 'num_object_copies': 17264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8234, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1398, 'num_bytes_recovered': 11419648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:26.362 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8018", 'reported_seq': 8098, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:23.186361+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:23.186361+0000', 'last_peered': '2024-09-15T21:09:23.186361+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:23.186361+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8018, 'log_dups_size': 0, 'ondisk_log_size': 8018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36158110, 'num_objects': 4415, 'num_object_clones': 0, 'num_object_copies': 17660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4415, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8017, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2157}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8141", 'reported_seq': 8211, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.478136+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:25.478136+0000', 'last_peered': '2024-09-15T21:09:25.478136+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:25.478136+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8141, 'log_dups_size': 0, 'ondisk_log_size': 8141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35345676, 'num_objects': 4315, 'num_object_clones': 0, 'num_object_copies': 17260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8139, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1981}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8156", 'reported_seq': 8210, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:21.037299+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:21.037299+0000', 'last_peered': '2024-09-15T21:09:21.037299+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:21.037299+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8156, 'log_dups_size': 0, 'ondisk_log_size': 8156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36885772, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 18012, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8154, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2192}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8008", 'reported_seq': 8053, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:21.164960+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:21.164960+0000', 'last_peered': '2024-09-15T21:09:21.164960+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:21.164960+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8008, 'log_dups_size': 0, 'ondisk_log_size': 8008, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35782656, 'num_objects': 4368, 'num_object_clones': 0, 'num_object_copies': 17472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8008, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2052}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:26.363 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8328", 'reported_seq': 8399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.514233+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:25.514233+0000', 'last_peered': '2024-09-15T21:09:25.514233+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:25.514233+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8328, 'log_dups_size': 0, 'ondisk_log_size': 8328, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36402444, 'num_objects': 4444, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8326, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2078}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8387", 'reported_seq': 8455, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.455088+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:25.455088+0000', 'last_peered': '2024-09-15T21:09:25.455088+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:25.455088+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8387, 'log_dups_size': 0, 'ondisk_log_size': 8387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35829004, 'num_objects': 4374, 'num_object_clones': 0, 'num_object_copies': 17496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8385, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8076", 'reported_seq': 8136, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:21.151179+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:21.151179+0000', 'last_peered': '2024-09-15T21:09:21.151179+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:21.151179+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8076, 'log_dups_size': 0, 'ondisk_log_size': 8076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35842584, 'num_objects': 4376, 'num_object_clones': 0, 'num_object_copies': 17504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4376, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8073, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 2044}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8131", 'reported_seq': 8213, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:23.058570+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:23.058570+0000', 'last_peered': '2024-09-15T21:09:23.058570+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:23.058570+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8131, 'log_dups_size': 0, 'ondisk_log_size': 8131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36295948, 'num_objects': 4431, 'num_object_clones': 0, 'num_object_copies': 17724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4431, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8129, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2096}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'7993", 'reported_seq': 8033, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:20.953686+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:20.953686+0000', 'last_peered': '2024-09-15T21:09:20.953686+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:20.953686+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 7993, 'log_dups_size': 0, 'ondisk_log_size': 7993, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36321926, 'num_objects': 4434, 'num_object_clones': 0, 'num_object_copies': 17736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7992, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2178}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:26.364 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8047", 'reported_seq': 8127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:23.210115+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:23.210115+0000', 'last_peered': '2024-09-15T21:09:23.210115+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:23.210115+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8047, 'log_dups_size': 0, 'ondisk_log_size': 8047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36485766, 'num_objects': 4454, 'num_object_clones': 0, 'num_object_copies': 17816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4454, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8046, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8000", 'reported_seq': 8057, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:20.988136+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:20.988136+0000', 'last_peered': '2024-09-15T21:09:20.988136+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:20.988136+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8000, 'log_dups_size': 0, 'ondisk_log_size': 8000, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36304140, 'num_objects': 4432, 'num_object_clones': 0, 'num_object_copies': 17728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 7998, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2153}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8119", 'reported_seq': 8202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:23.109036+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:23.109036+0000', 'last_peered': '2024-09-15T21:09:23.109036+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:23.109036+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8119, 'log_dups_size': 0, 'ondisk_log_size': 8119, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 37369100, 'num_objects': 4562, 'num_object_clones': 0, 'num_object_copies': 18248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4562, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8117, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2281}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:26.365 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8180", 'reported_seq': 8249, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.403484+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:25.403484+0000', 'last_peered': '2024-09-15T21:09:25.403484+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:25.403484+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8180, 'log_dups_size': 0, 'ondisk_log_size': 8180, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35788044, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8178, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:26.365 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:27.117 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:27.117 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33427/282252 objects degraded (11.843%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":432,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":70563,"data_bytes":578008878,"bytes_used":2791067648,"bytes_avail":383755988992,"bytes_total":386547056640,"degraded_objects":33427,"degraded_total":282252,"degraded_ratio":0.11842963025948443,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":99638,"recovering_keys_per_sec":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":235},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:27.140 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:27.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:27.777 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:27.778 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:27.822 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-15T21:09:26.052097+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":578008878,"num_objects":70563,"num_object_clones":0,"num_object_copies":282252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123849,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129762,"ondisk_log_size":129762,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2725652,"kb_used_data":1102928,"kb_used_omap":14,"kb_used_meta":1622641,"kb_avail":374761708,"statfs":{"total":386547056640,"available":383755988992,"internally_reserved":0,"allocated":1129398272,"data_stored":1128738316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1661584946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,6,17,8,14,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23109632,"num_objects":-2821,"num_object_clones":0,"num_object_copies":-11284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2821,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":146,"num_bytes_recovered":1196032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003740"},"pg_stats":[{"pgid":"1.f","version":"33'7954","reported_seq":7999,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.193624+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:21.193624+0000","last_peered":"2024-09-15T21:09:21.193624+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:21.193624+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7954,"log_dups_size":0,"ondisk_log_size":7954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36028416,"num_objects":4398,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7954,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8236","reported_seq":11097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.494769+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:25.494769+0000","last_peered":"2024-09-15T21:09:25.494769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:25.494769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35353868,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8018","reported_seq":8098,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.186361+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:23.186361+0000","last_peered":"2024-09-15T21:09:23.186361+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:23.186361+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8018,"log_dups_size":0,"ondisk_log_size":8018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36158110,"num_objects":4415,"num_object_clones":0,"num_object_copies":17660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8017,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2157}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8141","reported_seq":8211,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.478136+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:25.478136+0000","last_peered":"2024-09-15T21:09:25.478136+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:25.478136+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8156","reported_seq":8210,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.037299+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:21.037299+0000","last_peered":"2024-09-15T21:09:21.037299+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:21.037299+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36885772,"num_objects":4503,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8154,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2192}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8008","reported_seq":8053,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.164960+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:21.164960+0000","last_peered":"2024-09-15T21:09:21.164960+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:21.164960+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8008,"log_dups_size":0,"ondisk_log_size":8008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35782656,"num_objects":4368,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2052}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8328","reported_seq":8399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.514233+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:25.514233+0000","last_peered":"2024-09-15T21:09:25.514233+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:25.514233+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402444,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8387","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.455088+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:25.455088+0000","last_peered":"2024-09-15T21:09:25.455088+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:25.455088+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35829004,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8385,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8076","reported_seq":8136,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:21.151179+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:21.151179+0000","last_peered":"2024-09-15T21:09:21.151179+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:21.151179+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8076,"log_dups_size":0,"ondisk_log_size":8076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35842584,"num_objects":4376,"num_object_clones":0,"num_object_copies":17504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8073,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8131","reported_seq":8213,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.058570+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:23.058570+0000","last_peered":"2024-09-15T21:09:23.058570+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:23.058570+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8131,"log_dups_size":0,"ondisk_log_size":8131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36295948,"num_objects":4431,"num_object_clones":0,"num_object_copies":17724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8129,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'7988","reported_seq":12651,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:21.172568+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:21.172568+0000","last_peered":"2024-09-15T21:09:21.172568+0000","last_clean":"2024-09-15T21:09:21.172568+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:21.172568+0000","last_undegraded":"2024-09-15T21:09:21.172568+0000","last_fullsized":"2024-09-15T21:09:21.172568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35815424,"num_objects":4372,"num_object_clones":0,"num_object_copies":17488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'7993","reported_seq":8033,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.953686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:20.953686+0000","last_peered":"2024-09-15T21:09:20.953686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:20.953686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36321926,"num_objects":4434,"num_object_clones":0,"num_object_copies":17736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7992,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2178}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8047","reported_seq":8127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.210115+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:23.210115+0000","last_peered":"2024-09-15T21:09:23.210115+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:23.210115+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8047,"log_dups_size":0,"ondisk_log_size":8047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36485766,"num_objects":4454,"num_object_clones":0,"num_object_copies":17816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8046,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2168}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8000","reported_seq":8057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:20.988136+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:20.988136+0000","last_peered":"2024-09-15T21:09:20.988136+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:20.988136+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8000,"log_dups_size":0,"ondisk_log_size":8000,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36304140,"num_objects":4432,"num_object_clones":0,"num_object_copies":17728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7998,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2153}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8119","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:23.109036+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:23.109036+0000","last_peered":"2024-09-15T21:09:23.109036+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:23.109036+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8119,"log_dups_size":0,"ondisk_log_size":8119,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37369100,"num_objects":4562,"num_object_clones":0,"num_object_copies":18248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8117,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2281}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8180","reported_seq":8249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.403484+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:25.403484+0000","last_peered":"2024-09-15T21:09:25.403484+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:25.403484+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":578008878,"num_objects":70563,"num_object_clones":0,"num_object_copies":282252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70563,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":123849,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1127116800,"data_stored":1127116800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129762,"ondisk_log_size":129762,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717444,"kb_used_data":284532,"kb_used_omap":4,"kb_used_meta":432891,"kb_avail":93654396,"statfs":{"total":96636764160,"available":95902101504,"internally_reserved":0,"allocated":291360768,"data_stored":291195779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":443281104},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719696,"kb_used_data":283136,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93652144,"statfs":{"total":96636764160,"available":95899795456,"internally_reserved":0,"allocated":289931264,"data_stored":289766275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,2,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051650,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":561508,"kb_used_data":254996,"kb_used_omap":4,"kb_used_meta":306491,"kb_avail":93810332,"statfs":{"total":96636764160,"available":96061779968,"internally_reserved":0,"allocated":261115904,"data_stored":260950915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":313847503},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727004,"kb_used_data":280264,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93644836,"statfs":{"total":96636764160,"available":95892312064,"internally_reserved":0,"allocated":286990336,"data_stored":286825347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":457436878},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286568448,"data_stored":286568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260808704,"data_stored":260808704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":288882688,"data_stored":288882688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":290856960,"data_stored":290856960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:30.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:31.446 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:31.447 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:31.471 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-15T21:09:30.053586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571266862,"num_objects":69740,"num_object_clones":0,"num_object_copies":278960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751968,"kb_used_data":1093144,"kb_used_omap":14,"kb_used_meta":1658737,"kb_avail":374735392,"statfs":{"total":386547056640,"available":383729041408,"internally_reserved":0,"allocated":1119379456,"data_stored":1118719500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1698547250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,12,12,17,7,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22429696,"num_objects":-2738,"num_object_clones":0,"num_object_copies":-10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003929"},"pg_stats":[{"pgid":"1.f","version":"33'8020","reported_seq":8066,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.056773+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:26.056773+0000","last_peered":"2024-09-15T21:09:26.056773+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:26.056773+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35487744,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8236","reported_seq":11097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.494769+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:25.494769+0000","last_peered":"2024-09-15T21:09:25.494769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:25.494769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35353868,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8100","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.267865+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:28.267865+0000","last_peered":"2024-09-15T21:09:28.267865+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:28.267865+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8100,"log_dups_size":0,"ondisk_log_size":8100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8099,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8141","reported_seq":8211,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.478136+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:25.478136+0000","last_peered":"2024-09-15T21:09:25.478136+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:25.478136+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8236","reported_seq":8293,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.212239+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:26.212239+0000","last_peered":"2024-09-15T21:09:26.212239+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:26.212239+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230412,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8072","reported_seq":8117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.119236+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:26.119236+0000","last_peered":"2024-09-15T21:09:26.119236+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:26.119236+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8328","reported_seq":8399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.514233+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:25.514233+0000","last_peered":"2024-09-15T21:09:25.514233+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:25.514233+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402444,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8387","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.455088+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:25.455088+0000","last_peered":"2024-09-15T21:09:25.455088+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:25.455088+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35829004,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8385,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8152","reported_seq":8212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.229687+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:26.229687+0000","last_peered":"2024-09-15T21:09:26.229687+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:26.229687+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219992,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8149,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8191","reported_seq":8273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.200827+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:28.200827+0000","last_peered":"2024-09-15T21:09:28.200827+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:28.200827+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804428,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8189,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8051","reported_seq":12715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:26.250963+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:26.250963+0000","last_peered":"2024-09-15T21:09:26.250963+0000","last_clean":"2024-09-15T21:09:26.250963+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:26.250963+0000","last_undegraded":"2024-09-15T21:09:26.250963+0000","last_fullsized":"2024-09-15T21:09:26.250963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8051,"log_dups_size":0,"ondisk_log_size":8051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35299328,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8079","reported_seq":8119,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.266727+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:26.266727+0000","last_peered":"2024-09-15T21:09:26.266727+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:26.266727+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8079,"log_dups_size":0,"ondisk_log_size":8079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35617414,"num_objects":4348,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8078,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2092}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8128","reported_seq":8208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.248037+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:28.248037+0000","last_peered":"2024-09-15T21:09:28.248037+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:28.248037+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35822214,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8127,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2087}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8078","reported_seq":8135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.135849+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:26.135849+0000","last_peered":"2024-09-15T21:09:26.135849+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:26.135849+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35665164,"num_objects":4354,"num_object_clones":0,"num_object_copies":17416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8206","reported_seq":8288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.186041+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:28.186041+0000","last_peered":"2024-09-15T21:09:28.186041+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:28.186041+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36656396,"num_objects":4475,"num_object_clones":0,"num_object_copies":17900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8180","reported_seq":8249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.403484+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:25.403484+0000","last_peered":"2024-09-15T21:09:25.403484+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:25.403484+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571266862,"num_objects":69740,"num_object_clones":0,"num_object_copies":278960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1116958720,"data_stored":1116958720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734900,"kb_used_data":279776,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93636940,"statfs":{"total":96636764160,"available":95884226560,"internally_reserved":0,"allocated":286490624,"data_stored":286325635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":466022096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724632,"kb_used_data":278344,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93647208,"statfs":{"total":96636764160,"available":95894740992,"internally_reserved":0,"allocated":285024256,"data_stored":284859267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565432,"kb_used_data":254760,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93806408,"statfs":{"total":96636764160,"available":96057761792,"internally_reserved":0,"allocated":260874240,"data_stored":260709251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727004,"kb_used_data":280264,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93644836,"statfs":{"total":96636764160,"available":95892312064,"internally_reserved":0,"allocated":286990336,"data_stored":286825347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":457436878},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286568448,"data_stored":286568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260546560,"data_stored":260546560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283951104,"data_stored":283951104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285892608,"data_stored":285892608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:31.472 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:32.032 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:32.033 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:32.065 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-15T21:09:30.053586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":571266862,"num_objects":69740,"num_object_clones":0,"num_object_copies":278960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2751968,"kb_used_data":1093144,"kb_used_omap":14,"kb_used_meta":1658737,"kb_avail":374735392,"statfs":{"total":386547056640,"available":383729041408,"internally_reserved":0,"allocated":1119379456,"data_stored":1118719500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1698547250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,12,12,17,7,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22429696,"num_objects":-2738,"num_object_clones":0,"num_object_copies":-10952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2738,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":786432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003929"},"pg_stats":[{"pgid":"1.f","version":"33'8020","reported_seq":8066,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.056773+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:26.056773+0000","last_peered":"2024-09-15T21:09:26.056773+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:26.056773+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8020,"log_dups_size":0,"ondisk_log_size":8020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35487744,"num_objects":4332,"num_object_clones":0,"num_object_copies":17328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8020,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2056}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8236","reported_seq":11097,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.494769+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:25.494769+0000","last_peered":"2024-09-15T21:09:25.494769+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:25.494769+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35353868,"num_objects":4316,"num_object_clones":0,"num_object_copies":17264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":986,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8100","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.267865+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:28.267865+0000","last_peered":"2024-09-15T21:09:28.267865+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:28.267865+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8100,"log_dups_size":0,"ondisk_log_size":8100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8099,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8141","reported_seq":8211,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.478136+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:25.478136+0000","last_peered":"2024-09-15T21:09:25.478136+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:25.478136+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35345676,"num_objects":4315,"num_object_clones":0,"num_object_copies":17260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8139,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8236","reported_seq":8293,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.212239+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:26.212239+0000","last_peered":"2024-09-15T21:09:26.212239+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:26.212239+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36230412,"num_objects":4423,"num_object_clones":0,"num_object_copies":17692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8072","reported_seq":8117,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.119236+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:26.119236+0000","last_peered":"2024-09-15T21:09:26.119236+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:26.119236+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8072,"log_dups_size":0,"ondisk_log_size":8072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":4304,"num_object_clones":0,"num_object_copies":17216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1988}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8328","reported_seq":8399,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.514233+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:25.514233+0000","last_peered":"2024-09-15T21:09:25.514233+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:25.514233+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8328,"log_dups_size":0,"ondisk_log_size":8328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36402444,"num_objects":4444,"num_object_clones":0,"num_object_copies":17776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8326,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":2078}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8387","reported_seq":8455,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.455088+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:25.455088+0000","last_peered":"2024-09-15T21:09:25.455088+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:25.455088+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35829004,"num_objects":4374,"num_object_clones":0,"num_object_copies":17496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8385,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8152","reported_seq":8212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.229687+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:26.229687+0000","last_peered":"2024-09-15T21:09:26.229687+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:26.229687+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35219992,"num_objects":4300,"num_object_clones":0,"num_object_copies":17200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8149,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8191","reported_seq":8273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.200827+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:28.200827+0000","last_peered":"2024-09-15T21:09:28.200827+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:28.200827+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804428,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8189,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8051","reported_seq":12715,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:26.250963+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:26.250963+0000","last_peered":"2024-09-15T21:09:26.250963+0000","last_clean":"2024-09-15T21:09:26.250963+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:26.250963+0000","last_undegraded":"2024-09-15T21:09:26.250963+0000","last_fullsized":"2024-09-15T21:09:26.250963+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8051,"log_dups_size":0,"ondisk_log_size":8051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35299328,"num_objects":4309,"num_object_clones":0,"num_object_copies":17236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8079","reported_seq":8119,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.266727+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:26.266727+0000","last_peered":"2024-09-15T21:09:26.266727+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:26.266727+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8079,"log_dups_size":0,"ondisk_log_size":8079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35617414,"num_objects":4348,"num_object_clones":0,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8078,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2092}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8128","reported_seq":8208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.248037+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:28.248037+0000","last_peered":"2024-09-15T21:09:28.248037+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:28.248037+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35822214,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8127,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2087}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8078","reported_seq":8135,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:26.135849+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:26.135849+0000","last_peered":"2024-09-15T21:09:26.135849+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:26.135849+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8078,"log_dups_size":0,"ondisk_log_size":8078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35665164,"num_objects":4354,"num_object_clones":0,"num_object_copies":17416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2075}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8206","reported_seq":8288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.186041+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:28.186041+0000","last_peered":"2024-09-15T21:09:28.186041+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:28.186041+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36656396,"num_objects":4475,"num_object_clones":0,"num_object_copies":17900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8180","reported_seq":8249,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:25.403484+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:25.403484+0000","last_peered":"2024-09-15T21:09:25.403484+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:25.403484+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35788044,"num_objects":4369,"num_object_clones":0,"num_object_copies":17476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8178,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":2054}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":571266862,"num_objects":69740,"num_object_clones":0,"num_object_copies":278960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1420,"num_bytes_recovered":11419648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1116958720,"data_stored":1116958720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130585,"ondisk_log_size":130585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734900,"kb_used_data":279776,"kb_used_omap":4,"kb_used_meta":455099,"kb_avail":93636940,"statfs":{"total":96636764160,"available":95884226560,"internally_reserved":0,"allocated":286490624,"data_stored":286325635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":466022096},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,0,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724632,"kb_used_data":278344,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93647208,"statfs":{"total":96636764160,"available":95894740992,"internally_reserved":0,"allocated":285024256,"data_stored":284859267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565432,"kb_used_data":254760,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93806408,"statfs":{"total":96636764160,"available":96057761792,"internally_reserved":0,"allocated":260874240,"data_stored":260709251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247888,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727004,"kb_used_data":280264,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93644836,"statfs":{"total":96636764160,"available":95892312064,"internally_reserved":0,"allocated":286990336,"data_stored":286825347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":457436878},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,7,2,1,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":286568448,"data_stored":286568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260546560,"data_stored":260546560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283951104,"data_stored":283951104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":285892608,"data_stored":285892608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:32.066 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:32.066 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8020", 'reported_seq': 8066, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.056773+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:26.056773+0000', 'last_peered': '2024-09-15T21:09:26.056773+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:26.056773+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8020, 'log_dups_size': 0, 'ondisk_log_size': 8020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35487744, 'num_objects': 4332, 'num_object_clones': 0, 'num_object_copies': 17328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4332, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8020, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2056}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8236", 'reported_seq': 11097, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.494769+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:25.494769+0000', 'last_peered': '2024-09-15T21:09:25.494769+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:25.494769+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8236, 'log_dups_size': 0, 'ondisk_log_size': 8236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35353868, 'num_objects': 4316, 'num_object_clones': 0, 'num_object_copies': 17264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 986, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8234, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1398, 'num_bytes_recovered': 11419648, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8100", 'reported_seq': 8178, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:28.267865+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:28.267865+0000', 'last_peered': '2024-09-15T21:09:28.267865+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:28.267865+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8100, 'log_dups_size': 0, 'ondisk_log_size': 8100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35486366, 'num_objects': 4333, 'num_object_clones': 0, 'num_object_copies': 17332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4333, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8099, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8141", 'reported_seq': 8211, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.478136+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:25.478136+0000', 'last_peered': '2024-09-15T21:09:25.478136+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:25.478136+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8141, 'log_dups_size': 0, 'ondisk_log_size': 8141, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35345676, 'num_objects': 4315, 'num_object_clones': 0, 'num_object_copies': 17260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8139, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1981}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8236", 'reported_seq': 8293, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.212239+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:26.212239+0000', 'last_peered': '2024-09-15T21:09:26.212239+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:26.212239+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8236, 'log_dups_size': 0, 'ondisk_log_size': 8236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36230412, 'num_objects': 4423, 'num_object_clones': 0, 'num_object_copies': 17692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4423, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8234, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2112}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8072", 'reported_seq': 8117, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.119236+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:26.119236+0000', 'last_peered': '2024-09-15T21:09:26.119236+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:26.119236+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8072, 'log_dups_size': 0, 'ondisk_log_size': 8072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35258368, 'num_objects': 4304, 'num_object_clones': 0, 'num_object_copies': 17216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4304, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8072, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1988}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.067 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8328", 'reported_seq': 8399, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.514233+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:25.514233+0000', 'last_peered': '2024-09-15T21:09:25.514233+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:25.514233+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8328, 'log_dups_size': 0, 'ondisk_log_size': 8328, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36402444, 'num_objects': 4444, 'num_object_clones': 0, 'num_object_copies': 17776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4444, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8326, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 2078}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8387", 'reported_seq': 8455, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.455088+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:25.455088+0000', 'last_peered': '2024-09-15T21:09:25.455088+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:25.455088+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8387, 'log_dups_size': 0, 'ondisk_log_size': 8387, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35829004, 'num_objects': 4374, 'num_object_clones': 0, 'num_object_copies': 17496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4374, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8385, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8152", 'reported_seq': 8212, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.229687+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:26.229687+0000', 'last_peered': '2024-09-15T21:09:26.229687+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:26.229687+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8152, 'log_dups_size': 0, 'ondisk_log_size': 8152, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35219992, 'num_objects': 4300, 'num_object_clones': 0, 'num_object_copies': 17200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4300, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8149, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1968}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8191", 'reported_seq': 8273, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:28.200827+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:28.200827+0000', 'last_peered': '2024-09-15T21:09:28.200827+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:28.200827+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8191, 'log_dups_size': 0, 'ondisk_log_size': 8191, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35804428, 'num_objects': 4371, 'num_object_clones': 0, 'num_object_copies': 17484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4371, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8189, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 2036}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8079", 'reported_seq': 8119, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.266727+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:26.266727+0000', 'last_peered': '2024-09-15T21:09:26.266727+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:26.266727+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8079, 'log_dups_size': 0, 'ondisk_log_size': 8079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35617414, 'num_objects': 4348, 'num_object_clones': 0, 'num_object_copies': 17392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8078, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2092}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:32.068 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8128", 'reported_seq': 8208, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:28.248037+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:28.248037+0000', 'last_peered': '2024-09-15T21:09:28.248037+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:28.248037+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8128, 'log_dups_size': 0, 'ondisk_log_size': 8128, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35822214, 'num_objects': 4373, 'num_object_clones': 0, 'num_object_copies': 17492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8127, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2087}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8078", 'reported_seq': 8135, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:26.135849+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:26.135849+0000', 'last_peered': '2024-09-15T21:09:26.135849+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:26.135849+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8078, 'log_dups_size': 0, 'ondisk_log_size': 8078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35665164, 'num_objects': 4354, 'num_object_clones': 0, 'num_object_copies': 17416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4354, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8076, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8206", 'reported_seq': 8288, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:28.186041+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:28.186041+0000', 'last_peered': '2024-09-15T21:09:28.186041+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:28.186041+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8206, 'log_dups_size': 0, 'ondisk_log_size': 8206, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36656396, 'num_objects': 4475, 'num_object_clones': 0, 'num_object_copies': 17900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8204, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2194}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:32.069 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8180", 'reported_seq': 8249, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:25.403484+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:25.403484+0000', 'last_peered': '2024-09-15T21:09:25.403484+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:25.403484+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8180, 'log_dups_size': 0, 'ondisk_log_size': 8180, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35788044, 'num_objects': 4369, 'num_object_clones': 0, 'num_object_copies': 17476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8178, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 2054}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:32.069 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:32.838 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:32.838 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33370/275344 objects degraded (12.119%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":438,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":68836,"data_bytes":563861294,"bytes_used":2799652864,"bytes_avail":383747403776,"bytes_total":386547056640,"degraded_objects":33370,"degraded_total":275344,"degraded_ratio":0.12119385205415771,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":104414,"recovering_keys_per_sec":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":278},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:32.876 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:32.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:33.455 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:33.456 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:33.480 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-15T21:09:32.054246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":563861294,"num_objects":68836,"num_object_clones":0,"num_object_copies":275344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125576,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1477,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131489,"ondisk_log_size":131489,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2734036,"kb_used_data":1083516,"kb_used_omap":14,"kb_used_meta":1650417,"kb_avail":374753324,"statfs":{"total":386547056640,"available":383747403776,"internally_reserved":0,"allocated":1109520384,"data_stored":1108860428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1690027570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,1,7,10,14,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27385856,"num_objects":-3343,"num_object_clones":0,"num_object_copies":-13372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3343,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1253376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003883"},"pg_stats":[{"pgid":"1.f","version":"33'8111","reported_seq":8155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.280034+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:31.280034+0000","last_peered":"2024-09-15T21:09:31.280034+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:31.280034+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34742272,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8301","reported_seq":11276,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.507255+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:30.507255+0000","last_peered":"2024-09-15T21:09:30.507255+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:30.507255+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8301,"log_dups_size":0,"ondisk_log_size":8301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34821388,"num_objects":4251,"num_object_clones":0,"num_object_copies":17004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8299,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1455,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8100","reported_seq":8178,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.267865+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:28.267865+0000","last_peered":"2024-09-15T21:09:28.267865+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:28.267865+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8100,"log_dups_size":0,"ondisk_log_size":8100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35486366,"num_objects":4333,"num_object_clones":0,"num_object_copies":17332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4333,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8099,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2075}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8224","reported_seq":8294,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.578168+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:30.578168+0000","last_peered":"2024-09-15T21:09:30.578168+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:30.578168+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34665740,"num_objects":4232,"num_object_clones":0,"num_object_copies":16928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1898}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8306","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.248994+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:31.248994+0000","last_peered":"2024-09-15T21:09:31.248994+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:31.248994+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35656972,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8160","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269184+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:31.269184+0000","last_peered":"2024-09-15T21:09:31.269184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:31.269184+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8415","reported_seq":8485,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.624133+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:30.624133+0000","last_peered":"2024-09-15T21:09:30.624133+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:30.624133+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8415,"log_dups_size":0,"ondisk_log_size":8415,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35689740,"num_objects":4357,"num_object_clones":0,"num_object_copies":17428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1991}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8462","reported_seq":8532,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.452747+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:30.452747+0000","last_peered":"2024-09-15T21:09:30.452747+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:30.452747+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8462,"log_dups_size":0,"ondisk_log_size":8462,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35214604,"num_objects":4299,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4299,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8460,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1825}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8232","reported_seq":8292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269069+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:31.269069+0000","last_peered":"2024-09-15T21:09:31.269069+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:31.269069+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34564632,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8191","reported_seq":8273,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.200827+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:28.200827+0000","last_peered":"2024-09-15T21:09:28.200827+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:28.200827+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8191,"log_dups_size":0,"ondisk_log_size":8191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35804428,"num_objects":4371,"num_object_clones":0,"num_object_copies":17484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8189,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":2036}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8121","reported_seq":12785,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:31.139715+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:31.139715+0000","last_peered":"2024-09-15T21:09:31.139715+0000","last_clean":"2024-09-15T21:09:31.139715+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:31.139715+0000","last_undegraded":"2024-09-15T21:09:31.139715+0000","last_fullsized":"2024-09-15T21:09:31.139715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8121,"log_dups_size":0,"ondisk_log_size":8121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34725888,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8134","reported_seq":8174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.998004+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:30.998004+0000","last_peered":"2024-09-15T21:09:30.998004+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:30.998004+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8134,"log_dups_size":0,"ondisk_log_size":8134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166854,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8128","reported_seq":8208,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.248037+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:28.248037+0000","last_peered":"2024-09-15T21:09:28.248037+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:28.248037+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8128,"log_dups_size":0,"ondisk_log_size":8128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35822214,"num_objects":4373,"num_object_clones":0,"num_object_copies":17492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8127,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2087}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8145","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.072717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:31.072717+0000","last_peered":"2024-09-15T21:09:31.072717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:31.072717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35116300,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8143,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8206","reported_seq":8288,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:28.186041+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:28.186041+0000","last_peered":"2024-09-15T21:09:28.186041+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:28.186041+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8206,"log_dups_size":0,"ondisk_log_size":8206,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36656396,"num_objects":4475,"num_object_clones":0,"num_object_copies":17900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2194}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8253","reported_seq":8322,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.602800+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:30.602800+0000","last_peered":"2024-09-15T21:09:30.602800+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:30.602800+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8253,"log_dups_size":0,"ondisk_log_size":8253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35190028,"num_objects":4296,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1981}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":563861294,"num_objects":68836,"num_object_clones":0,"num_object_copies":275344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33370,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":125576,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1477,"num_bytes_recovered":11886592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107066880,"data_stored":1107066880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131489,"ondisk_log_size":131489,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717560,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93654280,"statfs":{"total":96636764160,"available":95901982720,"internally_reserved":0,"allocated":281493504,"data_stored":281328515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724632,"kb_used_data":278344,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":93647208,"statfs":{"total":96636764160,"available":95894740992,"internally_reserved":0,"allocated":285024256,"data_stored":284859267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051651,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565432,"kb_used_data":254760,"kb_used_omap":4,"kb_used_meta":310651,"kb_avail":93806408,"statfs":{"total":96636764160,"available":96057761792,"internally_reserved":0,"allocated":260874240,"data_stored":260709251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":318107343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,4,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247889,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726412,"kb_used_data":275516,"kb_used_omap":4,"kb_used_meta":450875,"kb_avail":93645428,"statfs":{"total":96636764160,"available":95892918272,"internally_reserved":0,"allocated":282128384,"data_stored":281963395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":461696718},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,4,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281624576,"data_stored":281624576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260546560,"data_stored":260546560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":283951104,"data_stored":283951104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280944640,"data_stored":280944640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:36.485 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:37.121 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:37.121 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:37.149 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-15T21:09:36.055900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558511918,"num_objects":68183,"num_object_clones":0,"num_object_copies":272732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126229,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132142,"ondisk_log_size":132142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720188,"kb_used_data":1073808,"kb_used_omap":14,"kb_used_meta":1646257,"kb_avail":374767172,"statfs":{"total":386547056640,"available":383761584128,"internally_reserved":0,"allocated":1099579392,"data_stored":1098919436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1685767730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,11,18,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22765568,"num_objects":-2779,"num_object_clones":0,"num_object_copies":-11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004431"},"pg_stats":[{"pgid":"1.f","version":"33'8111","reported_seq":8155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.280034+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:31.280034+0000","last_peered":"2024-09-15T21:09:31.280034+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:31.280034+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34742272,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8375","reported_seq":11467,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.655872+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:35.655872+0000","last_peered":"2024-09-15T21:09:35.655872+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:35.655872+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8151","reported_seq":8232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.143216+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:33.143216+0000","last_peered":"2024-09-15T21:09:33.143216+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:33.143216+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068574,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8150,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8278","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.396505+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:35.396505+0000","last_peered":"2024-09-15T21:09:35.396505+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:35.396505+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223372,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8306","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.248994+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:31.248994+0000","last_peered":"2024-09-15T21:09:31.248994+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:31.248994+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35656972,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8160","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269184+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:31.269184+0000","last_peered":"2024-09-15T21:09:31.269184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:31.269184+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8492","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.553170+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:35.553170+0000","last_peered":"2024-09-15T21:09:35.553170+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:35.553170+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1914}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8541","reported_seq":8611,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.417773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:35.417773+0000","last_peered":"2024-09-15T21:09:35.417773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:35.417773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8232","reported_seq":8292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269069+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:31.269069+0000","last_peered":"2024-09-15T21:09:31.269069+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:31.269069+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34564632,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8282","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259142+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:33.259142+0000","last_peered":"2024-09-15T21:09:33.259142+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:33.259142+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8121","reported_seq":12785,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:31.139715+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:31.139715+0000","last_peered":"2024-09-15T21:09:31.139715+0000","last_clean":"2024-09-15T21:09:31.139715+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:31.139715+0000","last_undegraded":"2024-09-15T21:09:31.139715+0000","last_fullsized":"2024-09-15T21:09:31.139715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8121,"log_dups_size":0,"ondisk_log_size":8121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34725888,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8134","reported_seq":8174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.998004+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:30.998004+0000","last_peered":"2024-09-15T21:09:30.998004+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:30.998004+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8134,"log_dups_size":0,"ondisk_log_size":8134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166854,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8217","reported_seq":8298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259030+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:33.259030+0000","last_peered":"2024-09-15T21:09:33.259030+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:33.259030+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8216,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8145","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.072717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:31.072717+0000","last_peered":"2024-09-15T21:09:31.072717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:31.072717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35116300,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8143,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8274","reported_seq":8358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.255635+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:33.255635+0000","last_peered":"2024-09-15T21:09:33.255635+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:33.255635+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36099340,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8323","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.490764+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:35.490764+0000","last_peered":"2024-09-15T21:09:35.490764+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:35.490764+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8321,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558511918,"num_objects":68183,"num_object_clones":0,"num_object_copies":272732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126229,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097224192,"data_stored":1097224192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132142,"ondisk_log_size":132142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717560,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93654280,"statfs":{"total":96636764160,"available":95901982720,"internally_reserved":0,"allocated":281493504,"data_stored":281328515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723924,"kb_used_data":273464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93647916,"statfs":{"total":96636764160,"available":95895465984,"internally_reserved":0,"allocated":280027136,"data_stored":279862147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,12,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552688,"kb_used_data":254496,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93819152,"statfs":{"total":96636764160,"available":96070811648,"internally_reserved":0,"allocated":260603904,"data_stored":260438915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305327823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726016,"kb_used_data":270952,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93645824,"statfs":{"total":96636764160,"available":95893323776,"internally_reserved":0,"allocated":277454848,"data_stored":277289859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276848640,"data_stored":276848640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260292608,"data_stored":260292608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":279138304,"data_stored":279138304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280944640,"data_stored":280944640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:37.150 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:37.772 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:37.773 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:37.796 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-15T21:09:36.055900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":558511918,"num_objects":68183,"num_object_clones":0,"num_object_copies":272732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126229,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132142,"ondisk_log_size":132142,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2720188,"kb_used_data":1073808,"kb_used_omap":14,"kb_used_meta":1646257,"kb_avail":374767172,"statfs":{"total":386547056640,"available":383761584128,"internally_reserved":0,"allocated":1099579392,"data_stored":1098919436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1685767730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,4,11,18,11,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22765568,"num_objects":-2779,"num_object_clones":0,"num_object_copies":-11116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":1392640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004431"},"pg_stats":[{"pgid":"1.f","version":"33'8111","reported_seq":8155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.280034+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:31.280034+0000","last_peered":"2024-09-15T21:09:31.280034+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:31.280034+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8111,"log_dups_size":0,"ondisk_log_size":8111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34742272,"num_objects":4241,"num_object_clones":0,"num_object_copies":16964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1965}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8375","reported_seq":11467,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.655872+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:35.655872+0000","last_peered":"2024-09-15T21:09:35.655872+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:35.655872+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8151","reported_seq":8232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.143216+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:33.143216+0000","last_peered":"2024-09-15T21:09:33.143216+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:33.143216+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068574,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8150,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8278","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.396505+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:35.396505+0000","last_peered":"2024-09-15T21:09:35.396505+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:35.396505+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223372,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8306","reported_seq":8363,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.248994+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:31.248994+0000","last_peered":"2024-09-15T21:09:31.248994+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:31.248994+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35656972,"num_objects":4353,"num_object_clones":0,"num_object_copies":17412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8304,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":2042}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8160","reported_seq":8205,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269184+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:31.269184+0000","last_peered":"2024-09-15T21:09:31.269184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:31.269184+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8160,"log_dups_size":0,"ondisk_log_size":8160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8160,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1900}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8492","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.553170+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:35.553170+0000","last_peered":"2024-09-15T21:09:35.553170+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:35.553170+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1914}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8541","reported_seq":8611,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.417773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:35.417773+0000","last_peered":"2024-09-15T21:09:35.417773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:35.417773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8232","reported_seq":8292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.269069+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:31.269069+0000","last_peered":"2024-09-15T21:09:31.269069+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:31.269069+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34564632,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8229,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8282","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259142+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:33.259142+0000","last_peered":"2024-09-15T21:09:33.259142+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:33.259142+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8121","reported_seq":12785,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:31.139715+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:31.139715+0000","last_peered":"2024-09-15T21:09:31.139715+0000","last_clean":"2024-09-15T21:09:31.139715+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:31.139715+0000","last_undegraded":"2024-09-15T21:09:31.139715+0000","last_fullsized":"2024-09-15T21:09:31.139715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8121,"log_dups_size":0,"ondisk_log_size":8121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34725888,"num_objects":4239,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8134","reported_seq":8174,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:30.998004+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:30.998004+0000","last_peered":"2024-09-15T21:09:30.998004+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:30.998004+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8134,"log_dups_size":0,"ondisk_log_size":8134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35166854,"num_objects":4293,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8133,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2037}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8217","reported_seq":8298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259030+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:33.259030+0000","last_peered":"2024-09-15T21:09:33.259030+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:33.259030+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8216,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8145","reported_seq":8202,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:31.072717+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:31.072717+0000","last_peered":"2024-09-15T21:09:31.072717+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:31.072717+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8145,"log_dups_size":0,"ondisk_log_size":8145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35116300,"num_objects":4287,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8143,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2008}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8274","reported_seq":8358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.255635+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:33.255635+0000","last_peered":"2024-09-15T21:09:33.255635+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:33.255635+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36099340,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8323","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.490764+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:35.490764+0000","last_peered":"2024-09-15T21:09:35.490764+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:35.490764+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8321,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":558511918,"num_objects":68183,"num_object_clones":0,"num_object_copies":272732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68183,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126229,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097224192,"data_stored":1097224192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132142,"ondisk_log_size":132142,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717560,"kb_used_data":274896,"kb_used_omap":4,"kb_used_meta":442619,"kb_avail":93654280,"statfs":{"total":96636764160,"available":95901982720,"internally_reserved":0,"allocated":281493504,"data_stored":281328515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":453242576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1,4,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723924,"kb_used_data":273464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93647916,"statfs":{"total":96636764160,"available":95895465984,"internally_reserved":0,"allocated":280027136,"data_stored":279862147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,12,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552688,"kb_used_data":254496,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93819152,"statfs":{"total":96636764160,"available":96070811648,"internally_reserved":0,"allocated":260603904,"data_stored":260438915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305327823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726016,"kb_used_data":270952,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93645824,"statfs":{"total":96636764160,"available":95893323776,"internally_reserved":0,"allocated":277454848,"data_stored":277289859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276848640,"data_stored":276848640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260292608,"data_stored":260292608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":279138304,"data_stored":279138304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":280944640,"data_stored":280944640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:37.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:37.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8111", 'reported_seq': 8155, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:31.280034+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:31.280034+0000', 'last_peered': '2024-09-15T21:09:31.280034+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:31.280034+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8111, 'log_dups_size': 0, 'ondisk_log_size': 8111, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34742272, 'num_objects': 4241, 'num_object_clones': 0, 'num_object_copies': 16964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8111, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1965}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:37.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8375", 'reported_seq': 11467, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:35.655872+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:35.655872+0000', 'last_peered': '2024-09-15T21:09:35.655872+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:35.655872+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8375, 'log_dups_size': 0, 'ondisk_log_size': 8375, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34215180, 'num_objects': 4177, 'num_object_clones': 0, 'num_object_copies': 16708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 870, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4177, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8373, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1514, 'num_bytes_recovered': 12369920, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3307}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8151", 'reported_seq': 8232, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:33.143216+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:33.143216+0000', 'last_peered': '2024-09-15T21:09:33.143216+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:33.143216+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8151, 'log_dups_size': 0, 'ondisk_log_size': 8151, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35068574, 'num_objects': 4282, 'num_object_clones': 0, 'num_object_copies': 17128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4282, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8150, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 2024}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8278", 'reported_seq': 8348, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:35.396505+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:35.396505+0000', 'last_peered': '2024-09-15T21:09:35.396505+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:35.396505+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8278, 'log_dups_size': 0, 'ondisk_log_size': 8278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34223372, 'num_objects': 4178, 'num_object_clones': 0, 'num_object_copies': 16712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8276, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:37.799 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8306", 'reported_seq': 8363, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:31.248994+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:31.248994+0000', 'last_peered': '2024-09-15T21:09:31.248994+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:31.248994+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8306, 'log_dups_size': 0, 'ondisk_log_size': 8306, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35656972, 'num_objects': 4353, 'num_object_clones': 0, 'num_object_copies': 17412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4353, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8304, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 2042}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8160", 'reported_seq': 8205, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:31.269184+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:31.269184+0000', 'last_peered': '2024-09-15T21:09:31.269184+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:31.269184+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8160, 'log_dups_size': 0, 'ondisk_log_size': 8160, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34537472, 'num_objects': 4216, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8160, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1900}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8492", 'reported_seq': 8561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:35.553170+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:35.553170+0000', 'last_peered': '2024-09-15T21:09:35.553170+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:35.553170+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8492, 'log_dups_size': 0, 'ondisk_log_size': 8492, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35058956, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8490, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1914}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:37.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8541", 'reported_seq': 8611, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:35.417773+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:35.417773+0000', 'last_peered': '2024-09-15T21:09:35.417773+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:35.417773+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8541, 'log_dups_size': 0, 'ondisk_log_size': 8541, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34567436, 'num_objects': 4220, 'num_object_clones': 0, 'num_object_copies': 16880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8539, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1746}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8232", 'reported_seq': 8292, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:31.269069+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:31.269069+0000', 'last_peered': '2024-09-15T21:09:31.269069+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:31.269069+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8232, 'log_dups_size': 0, 'ondisk_log_size': 8232, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34564632, 'num_objects': 4220, 'num_object_clones': 0, 'num_object_copies': 16880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4220, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8229, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1888}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8282", 'reported_seq': 8364, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:33.259142+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:33.259142+0000', 'last_peered': '2024-09-15T21:09:33.259142+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:33.259142+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8282, 'log_dups_size': 0, 'ondisk_log_size': 8282, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35058956, 'num_objects': 4280, 'num_object_clones': 0, 'num_object_copies': 17120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4280, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8280, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1945}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:37.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8134", 'reported_seq': 8174, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:30.998004+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:30.998004+0000', 'last_peered': '2024-09-15T21:09:30.998004+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:30.998004+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8134, 'log_dups_size': 0, 'ondisk_log_size': 8134, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35166854, 'num_objects': 4293, 'num_object_clones': 0, 'num_object_copies': 17172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8133, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2037}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8217", 'reported_seq': 8298, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:33.259030+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:33.259030+0000', 'last_peered': '2024-09-15T21:09:33.259030+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:33.259030+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8217, 'log_dups_size': 0, 'ondisk_log_size': 8217, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35093126, 'num_objects': 4284, 'num_object_clones': 0, 'num_object_copies': 17136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4284, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8216, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1998}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8145", 'reported_seq': 8202, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:31.072717+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:31.072717+0000', 'last_peered': '2024-09-15T21:09:31.072717+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:31.072717+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8145, 'log_dups_size': 0, 'ondisk_log_size': 8145, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35116300, 'num_objects': 4287, 'num_object_clones': 0, 'num_object_copies': 17148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8143, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2008}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8274", 'reported_seq': 8358, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:33.255635+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:33.255635+0000', 'last_peered': '2024-09-15T21:09:33.255635+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:33.255635+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8274, 'log_dups_size': 0, 'ondisk_log_size': 8274, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 36099340, 'num_objects': 4407, 'num_object_clones': 0, 'num_object_copies': 17628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8272, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:37.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:37.803 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8323", 'reported_seq': 8392, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:35.490764+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:35.490764+0000', 'last_peered': '2024-09-15T21:09:35.490764+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:35.490764+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8323, 'log_dups_size': 0, 'ondisk_log_size': 8323, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34616588, 'num_objects': 4226, 'num_object_clones': 0, 'num_object_copies': 16904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8321, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:37.803 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:38.543 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:38.543 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33311/270624 objects degraded (12.309%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":444,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":67656,"data_bytes":554194734,"bytes_used":2803396608,"bytes_avail":383743660032,"bytes_total":386547056640,"degraded_objects":33311,"degraded_total":270624,"degraded_ratio":0.12308960033108668,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":79159,"recovering_keys_per_sec":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":242},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:38.566 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:38.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:39.173 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:39.174 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:39.207 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-15T21:09:38.056568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":554194734,"num_objects":67656,"num_object_clones":0,"num_object_copies":270624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126756,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132669,"ondisk_log_size":132669,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737692,"kb_used_data":1069128,"kb_used_omap":14,"kb_used_meta":1668465,"kb_avail":374749668,"statfs":{"total":386547056640,"available":383743660032,"internally_reserved":0,"allocated":1094787072,"data_stored":1094127116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1708508722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,2,11,21,13,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23814144,"num_objects":-2907,"num_object_clones":0,"num_object_copies":-11628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2907,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":950272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004470"},"pg_stats":[{"pgid":"1.f","version":"33'8181","reported_seq":8227,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.345141+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:36.345141+0000","last_peered":"2024-09-15T21:09:36.345141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:36.345141+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8181,"log_dups_size":0,"ondisk_log_size":8181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34168832,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1895}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8375","reported_seq":11467,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.655872+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:35.655872+0000","last_peered":"2024-09-15T21:09:35.655872+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:35.655872+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34215180,"num_objects":4177,"num_object_clones":0,"num_object_copies":16708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":870,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8373,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1514,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3307}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8151","reported_seq":8232,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.143216+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:33.143216+0000","last_peered":"2024-09-15T21:09:33.143216+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:33.143216+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8151,"log_dups_size":0,"ondisk_log_size":8151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35068574,"num_objects":4282,"num_object_clones":0,"num_object_copies":17128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8150,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":2024}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8278","reported_seq":8348,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.396505+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:35.396505+0000","last_peered":"2024-09-15T21:09:35.396505+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:35.396505+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8278,"log_dups_size":0,"ondisk_log_size":8278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34223372,"num_objects":4178,"num_object_clones":0,"num_object_copies":16712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8374","reported_seq":8430,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.372955+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:36.372955+0000","last_peered":"2024-09-15T21:09:36.372955+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:36.372955+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8374,"log_dups_size":0,"ondisk_log_size":8374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35099916,"num_objects":4285,"num_object_clones":0,"num_object_copies":17140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8372,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1974}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8239","reported_seq":8284,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.261328+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:36.261328+0000","last_peered":"2024-09-15T21:09:36.261328+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:36.261328+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8239,"log_dups_size":0,"ondisk_log_size":8239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33890304,"num_objects":4137,"num_object_clones":0,"num_object_copies":16548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8239,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8492","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.553170+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:35.553170+0000","last_peered":"2024-09-15T21:09:35.553170+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:35.553170+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8492,"log_dups_size":0,"ondisk_log_size":8492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8490,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1914}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8541","reported_seq":8611,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.417773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:35.417773+0000","last_peered":"2024-09-15T21:09:35.417773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:35.417773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8541,"log_dups_size":0,"ondisk_log_size":8541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34567436,"num_objects":4220,"num_object_clones":0,"num_object_copies":16880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8539,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8316","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.409990+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:36.409990+0000","last_peered":"2024-09-15T21:09:36.409990+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:36.409990+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8316,"log_dups_size":0,"ondisk_log_size":8316,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33876504,"num_objects":4136,"num_object_clones":0,"num_object_copies":16544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8313,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1804}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8282","reported_seq":8364,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259142+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:33.259142+0000","last_peered":"2024-09-15T21:09:33.259142+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:33.259142+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8282,"log_dups_size":0,"ondisk_log_size":8282,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35058956,"num_objects":4280,"num_object_clones":0,"num_object_copies":17120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1945}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8189","reported_seq":12853,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:36.285192+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:36.285192+0000","last_peered":"2024-09-15T21:09:36.285192+0000","last_clean":"2024-09-15T21:09:36.285192+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:36.285192+0000","last_undegraded":"2024-09-15T21:09:36.285192+0000","last_fullsized":"2024-09-15T21:09:36.285192+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8189,"log_dups_size":0,"ondisk_log_size":8189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34168832,"num_objects":4171,"num_object_clones":0,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8201","reported_seq":8241,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.446038+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:36.446038+0000","last_peered":"2024-09-15T21:09:36.446038+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:36.446038+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8201,"log_dups_size":0,"ondisk_log_size":8201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34617990,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8200,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1970}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8217","reported_seq":8298,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.259030+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:33.259030+0000","last_peered":"2024-09-15T21:09:33.259030+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:33.259030+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8217,"log_dups_size":0,"ondisk_log_size":8217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35093126,"num_objects":4284,"num_object_clones":0,"num_object_copies":17136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8216,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1998}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8236","reported_seq":8292,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:36.383673+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:36.383673+0000","last_peered":"2024-09-15T21:09:36.383673+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:36.383673+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8236,"log_dups_size":0,"ondisk_log_size":8236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34370828,"num_objects":4196,"num_object_clones":0,"num_object_copies":16784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8234,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1917}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8274","reported_seq":8358,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:33.255635+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:33.255635+0000","last_peered":"2024-09-15T21:09:33.255635+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:33.255635+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36099340,"num_objects":4407,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8323","reported_seq":8392,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:35.490764+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:35.490764+0000","last_peered":"2024-09-15T21:09:35.490764+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:35.490764+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34616588,"num_objects":4226,"num_object_clones":0,"num_object_copies":16904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8321,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":554194734,"num_objects":67656,"num_object_clones":0,"num_object_copies":270624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33311,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67656,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126756,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":12369920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1092272128,"data_stored":1092272128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132669,"ondisk_log_size":132669,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735064,"kb_used_data":270216,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93636776,"statfs":{"total":96636764160,"available":95884058624,"internally_reserved":0,"allocated":276701184,"data_stored":276536195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723924,"kb_used_data":273464,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":93647916,"statfs":{"total":96636764160,"available":95895465984,"internally_reserved":0,"allocated":280027136,"data_stored":279862147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,12,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051652,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552688,"kb_used_data":254496,"kb_used_omap":4,"kb_used_meta":298171,"kb_avail":93819152,"statfs":{"total":96636764160,"available":96070811648,"internally_reserved":0,"allocated":260603904,"data_stored":260438915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":305327823},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247890,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726016,"kb_used_data":270952,"kb_used_omap":4,"kb_used_meta":455035,"kb_avail":93645824,"statfs":{"total":96636764160,"available":95893323776,"internally_reserved":0,"allocated":277454848,"data_stored":277289859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":465956558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,6,5,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276848640,"data_stored":276848640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260292608,"data_stored":260292608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":279138304,"data_stored":279138304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275992576,"data_stored":275992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:42.210 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:42.849 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:42.850 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:42.875 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-15T21:09:42.058158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544765742,"num_objects":66505,"num_object_clones":0,"num_object_copies":266020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127907,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133820,"ondisk_log_size":133820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759540,"kb_used_data":1054900,"kb_used_omap":14,"kb_used_meta":1704561,"kb_avail":374727820,"statfs":{"total":386547056640,"available":383721287680,"internally_reserved":0,"allocated":1080217600,"data_stored":1079557644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1745471026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,13,13,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26501120,"num_objects":-3235,"num_object_clones":0,"num_object_copies":-12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004572"},"pg_stats":[{"pgid":"1.f","version":"33'8258","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.283549+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:41.283549+0000","last_peered":"2024-09-15T21:09:41.283549+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:41.283549+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33538048,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8448","reported_seq":11647,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.733973+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:40.733973+0000","last_peered":"2024-09-15T21:09:40.733973+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:40.733973+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33617164,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8224","reported_seq":8305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.328601+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:38.328601+0000","last_peered":"2024-09-15T21:09:38.328601+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:38.328601+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34470558,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8223,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1951}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8358","reported_seq":8427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.672885+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:40.672885+0000","last_peered":"2024-09-15T21:09:40.672885+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:40.672885+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8358,"log_dups_size":0,"ondisk_log_size":8358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8442","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.460040+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:41.460040+0000","last_peered":"2024-09-15T21:09:41.460040+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:41.460040+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542860,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8322","reported_seq":8367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.419745+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:41.419745+0000","last_peered":"2024-09-15T21:09:41.419745+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:41.419745+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8556","reported_seq":8627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.448221+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:40.448221+0000","last_peered":"2024-09-15T21:09:40.448221+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:40.448221+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534668,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8619","reported_seq":8689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.609759+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:40.609759+0000","last_peered":"2024-09-15T21:09:40.609759+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:40.609759+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8619,"log_dups_size":0,"ondisk_log_size":8619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928460,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8617,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8384","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.418095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:41.418095+0000","last_peered":"2024-09-15T21:09:41.418095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:41.418095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33319448,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8355","reported_seq":8437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.174600+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:38.174600+0000","last_peered":"2024-09-15T21:09:38.174600+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:38.174600+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8355,"log_dups_size":0,"ondisk_log_size":8355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34460940,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8244","reported_seq":12908,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:41.397256+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:41.397256+0000","last_peered":"2024-09-15T21:09:41.397256+0000","last_clean":"2024-09-15T21:09:41.397256+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:41.397256+0000","last_undegraded":"2024-09-15T21:09:41.397256+0000","last_fullsized":"2024-09-15T21:09:41.397256+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8264","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.293797+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:41.293797+0000","last_peered":"2024-09-15T21:09:41.293797+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:41.293797+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34101894,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8291","reported_seq":8372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.257964+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:38.257964+0000","last_peered":"2024-09-15T21:09:38.257964+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:38.257964+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8291,"log_dups_size":0,"ondisk_log_size":8291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34486918,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8290,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8304","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.352466+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:41.352466+0000","last_peered":"2024-09-15T21:09:41.352466+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:41.352466+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813772,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8360","reported_seq":8444,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.294802+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:38.294802+0000","last_peered":"2024-09-15T21:09:38.294802+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:38.294802+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394828,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8391","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.702445+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:40.702445+0000","last_peered":"2024-09-15T21:09:40.702445+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:40.702445+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059532,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":544765742,"num_objects":66505,"num_object_clones":0,"num_object_copies":266020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127907,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077555200,"data_stored":1077555200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133820,"ondisk_log_size":133820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734468,"kb_used_data":265460,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93637372,"statfs":{"total":96636764160,"available":95884668928,"internally_reserved":0,"allocated":271831040,"data_stored":271666051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,4,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741472,"kb_used_data":268816,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93630368,"statfs":{"total":96636764160,"available":95877496832,"internally_reserved":0,"allocated":275267584,"data_stored":275102595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,4,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556612,"kb_used_data":254260,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93815228,"statfs":{"total":96636764160,"available":96066793472,"internally_reserved":0,"allocated":260362240,"data_stored":260197251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726988,"kb_used_data":266364,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93644852,"statfs":{"total":96636764160,"available":95892328448,"internally_reserved":0,"allocated":272756736,"data_stored":272591747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271982592,"data_stored":271982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260079616,"data_stored":260079616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274268160,"data_stored":274268160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271224832,"data_stored":271224832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:42.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:43.481 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:43.482 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:43.508 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-15T21:09:42.058158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":544765742,"num_objects":66505,"num_object_clones":0,"num_object_copies":266020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127907,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133820,"ondisk_log_size":133820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2759540,"kb_used_data":1054900,"kb_used_omap":14,"kb_used_meta":1704561,"kb_avail":374727820,"statfs":{"total":386547056640,"available":383721287680,"internally_reserved":0,"allocated":1080217600,"data_stored":1079557644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1745471026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,13,13,5,12,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-26501120,"num_objects":-3235,"num_object_clones":0,"num_object_copies":-12940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3235,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004572"},"pg_stats":[{"pgid":"1.f","version":"33'8258","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.283549+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:41.283549+0000","last_peered":"2024-09-15T21:09:41.283549+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:41.283549+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33538048,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8448","reported_seq":11647,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.733973+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:40.733973+0000","last_peered":"2024-09-15T21:09:40.733973+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:40.733973+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33617164,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8224","reported_seq":8305,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.328601+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:38.328601+0000","last_peered":"2024-09-15T21:09:38.328601+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:38.328601+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8224,"log_dups_size":0,"ondisk_log_size":8224,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34470558,"num_objects":4209,"num_object_clones":0,"num_object_copies":16836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8223,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1951}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8358","reported_seq":8427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.672885+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:40.672885+0000","last_peered":"2024-09-15T21:09:40.672885+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:40.672885+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8358,"log_dups_size":0,"ondisk_log_size":8358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8442","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.460040+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:41.460040+0000","last_peered":"2024-09-15T21:09:41.460040+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:41.460040+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542860,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8322","reported_seq":8367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.419745+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:41.419745+0000","last_peered":"2024-09-15T21:09:41.419745+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:41.419745+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8556","reported_seq":8627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.448221+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:40.448221+0000","last_peered":"2024-09-15T21:09:40.448221+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:40.448221+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534668,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8619","reported_seq":8689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.609759+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:40.609759+0000","last_peered":"2024-09-15T21:09:40.609759+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:40.609759+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8619,"log_dups_size":0,"ondisk_log_size":8619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928460,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8617,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8384","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.418095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:41.418095+0000","last_peered":"2024-09-15T21:09:41.418095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:41.418095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33319448,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8355","reported_seq":8437,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.174600+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:38.174600+0000","last_peered":"2024-09-15T21:09:38.174600+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:38.174600+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8355,"log_dups_size":0,"ondisk_log_size":8355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34460940,"num_objects":4207,"num_object_clones":0,"num_object_copies":16828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8353,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1872}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8244","reported_seq":12908,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:41.397256+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:41.397256+0000","last_peered":"2024-09-15T21:09:41.397256+0000","last_clean":"2024-09-15T21:09:41.397256+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:41.397256+0000","last_undegraded":"2024-09-15T21:09:41.397256+0000","last_fullsized":"2024-09-15T21:09:41.397256+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8264","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.293797+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:41.293797+0000","last_peered":"2024-09-15T21:09:41.293797+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:41.293797+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34101894,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8291","reported_seq":8372,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.257964+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:38.257964+0000","last_peered":"2024-09-15T21:09:38.257964+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:38.257964+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8291,"log_dups_size":0,"ondisk_log_size":8291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34486918,"num_objects":4210,"num_object_clones":0,"num_object_copies":16840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8290,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1924}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8304","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.352466+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:41.352466+0000","last_peered":"2024-09-15T21:09:41.352466+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:41.352466+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813772,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8360","reported_seq":8444,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:38.294802+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:38.294802+0000","last_peered":"2024-09-15T21:09:38.294802+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:38.294802+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8360,"log_dups_size":0,"ondisk_log_size":8360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35394828,"num_objects":4321,"num_object_clones":0,"num_object_copies":17284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":2040}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8391","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.702445+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:40.702445+0000","last_peered":"2024-09-15T21:09:40.702445+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:40.702445+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059532,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":544765742,"num_objects":66505,"num_object_clones":0,"num_object_copies":266020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66505,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127907,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077555200,"data_stored":1077555200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133820,"ondisk_log_size":133820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734468,"kb_used_data":265460,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93637372,"statfs":{"total":96636764160,"available":95884668928,"internally_reserved":0,"allocated":271831040,"data_stored":271666051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,4,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741472,"kb_used_data":268816,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93630368,"statfs":{"total":96636764160,"available":95877496832,"internally_reserved":0,"allocated":275267584,"data_stored":275102595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,4,3,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556612,"kb_used_data":254260,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93815228,"statfs":{"total":96636764160,"available":96066793472,"internally_reserved":0,"allocated":260362240,"data_stored":260197251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726988,"kb_used_data":266364,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93644852,"statfs":{"total":96636764160,"available":95892328448,"internally_reserved":0,"allocated":272756736,"data_stored":272591747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271982592,"data_stored":271982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260079616,"data_stored":260079616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274268160,"data_stored":274268160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271224832,"data_stored":271224832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:43.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:43.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8258", 'reported_seq': 8304, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.283549+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:41.283549+0000', 'last_peered': '2024-09-15T21:09:41.283549+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:41.283549+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8258, 'log_dups_size': 0, 'ondisk_log_size': 8258, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33538048, 'num_objects': 4094, 'num_object_clones': 0, 'num_object_copies': 16376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4094, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8258, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1818}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.509 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:43.509 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8448", 'reported_seq': 11647, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:40.733973+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:40.733973+0000', 'last_peered': '2024-09-15T21:09:40.733973+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:40.733973+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8448, 'log_dups_size': 0, 'ondisk_log_size': 8448, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33617164, 'num_objects': 4104, 'num_object_clones': 0, 'num_object_copies': 16416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 817, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4104, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8446, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1567, 'num_bytes_recovered': 12804096, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3287}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8224", 'reported_seq': 8305, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:38.328601+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:38.328601+0000', 'last_peered': '2024-09-15T21:09:38.328601+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:38.328601+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8224, 'log_dups_size': 0, 'ondisk_log_size': 8224, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34470558, 'num_objects': 4209, 'num_object_clones': 0, 'num_object_copies': 16836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4209, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8223, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1951}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8358", 'reported_seq': 8427, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:40.672885+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:40.672885+0000', 'last_peered': '2024-09-15T21:09:40.672885+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:40.672885+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8358, 'log_dups_size': 0, 'ondisk_log_size': 8358, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33568012, 'num_objects': 4098, 'num_object_clones': 0, 'num_object_copies': 16392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8356, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1764}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8442", 'reported_seq': 8499, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.460040+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:41.460040+0000', 'last_peered': '2024-09-15T21:09:41.460040+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:41.460040+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8442, 'log_dups_size': 0, 'ondisk_log_size': 8442, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34542860, 'num_objects': 4217, 'num_object_clones': 0, 'num_object_copies': 16868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8440, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1906}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8322", 'reported_seq': 8367, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.419745+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:41.419745+0000', 'last_peered': '2024-09-15T21:09:41.419745+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:41.419745+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8322, 'log_dups_size': 0, 'ondisk_log_size': 8322, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33210368, 'num_objects': 4054, 'num_object_clones': 0, 'num_object_copies': 16216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4054, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8322, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1738}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:43.510 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8556", 'reported_seq': 8627, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:40.448221+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:40.448221+0000', 'last_peered': '2024-09-15T21:09:40.448221+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:40.448221+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8556, 'log_dups_size': 0, 'ondisk_log_size': 8556, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34534668, 'num_objects': 4216, 'num_object_clones': 0, 'num_object_copies': 16864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8554, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1850}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8619", 'reported_seq': 8689, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:40.609759+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:40.609759+0000', 'last_peered': '2024-09-15T21:09:40.609759+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:40.609759+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8619, 'log_dups_size': 0, 'ondisk_log_size': 8619, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33928460, 'num_objects': 4142, 'num_object_clones': 0, 'num_object_copies': 16568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8617, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1668}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8384", 'reported_seq': 8445, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.418095+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:41.418095+0000', 'last_peered': '2024-09-15T21:09:41.418095+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:41.418095+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8384, 'log_dups_size': 0, 'ondisk_log_size': 8384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33319448, 'num_objects': 4068, 'num_object_clones': 0, 'num_object_copies': 16272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8381, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1736}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8355", 'reported_seq': 8437, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:38.174600+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:38.174600+0000', 'last_peered': '2024-09-15T21:09:38.174600+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:38.174600+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8355, 'log_dups_size': 0, 'ondisk_log_size': 8355, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34460940, 'num_objects': 4207, 'num_object_clones': 0, 'num_object_copies': 16828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8353, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1872}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8264", 'reported_seq': 8304, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.293797+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:41.293797+0000', 'last_peered': '2024-09-15T21:09:41.293797+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:41.293797+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8264, 'log_dups_size': 0, 'ondisk_log_size': 8264, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34101894, 'num_objects': 4163, 'num_object_clones': 0, 'num_object_copies': 16652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4163, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8263, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1907}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8291", 'reported_seq': 8372, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:38.257964+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:38.257964+0000', 'last_peered': '2024-09-15T21:09:38.257964+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:38.257964+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8291, 'log_dups_size': 0, 'ondisk_log_size': 8291, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34486918, 'num_objects': 4210, 'num_object_clones': 0, 'num_object_copies': 16840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4210, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8290, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1924}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:43.511 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:43.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8304", 'reported_seq': 8360, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:41.352466+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:41.352466+0000', 'last_peered': '2024-09-15T21:09:41.352466+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:41.352466+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8304, 'log_dups_size': 0, 'ondisk_log_size': 8304, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33813772, 'num_objects': 4128, 'num_object_clones': 0, 'num_object_copies': 16512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8302, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1849}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:43.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:43.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8360", 'reported_seq': 8444, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:38.294802+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:38.294802+0000', 'last_peered': '2024-09-15T21:09:38.294802+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:38.294802+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8360, 'log_dups_size': 0, 'ondisk_log_size': 8360, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 35394828, 'num_objects': 4321, 'num_object_clones': 0, 'num_object_copies': 17284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8358, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 2040}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:43.512 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:43.512 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8391", 'reported_seq': 8460, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:40.702445+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:40.702445+0000', 'last_peered': '2024-09-15T21:09:40.702445+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:40.702445+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8391, 'log_dups_size': 0, 'ondisk_log_size': 8391, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34059532, 'num_objects': 4158, 'num_object_clones': 0, 'num_object_copies': 16632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4158, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8389, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:43.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:44.258 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:44.258 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33258/266020 objects degraded (12.502%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":450,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":66505,"data_bytes":544765742,"bytes_used":2825768960,"bytes_avail":383721287680,"bytes_total":386547056640,"degraded_objects":33258,"degraded_total":266020,"degraded_ratio":0.12502067513720772,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":115326,"recovering_keys_per_sec":0,"num_objects_recovered":169,"num_bytes_recovered":1384448,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":269},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:44.290 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:44.290 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:44.876 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:44.877 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:44.899 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-15T21:09:44.058706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":542070574,"num_objects":66176,"num_object_clones":0,"num_object_copies":264704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128236,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134149,"ondisk_log_size":134149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2758984,"kb_used_data":1050152,"kb_used_omap":14,"kb_used_meta":1708721,"kb_avail":374728376,"statfs":{"total":386547056640,"available":383721857024,"internally_reserved":0,"allocated":1075355648,"data_stored":1074695692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1749730866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,7,15,8,10,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21790720,"num_objects":-2660,"num_object_clones":0,"num_object_copies":-10640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2660,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":917504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004460"},"pg_stats":[{"pgid":"1.f","version":"33'8258","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.283549+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:41.283549+0000","last_peered":"2024-09-15T21:09:41.283549+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:41.283549+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8258,"log_dups_size":0,"ondisk_log_size":8258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33538048,"num_objects":4094,"num_object_clones":0,"num_object_copies":16376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8258,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1818}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8448","reported_seq":11647,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.733973+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:40.733973+0000","last_peered":"2024-09-15T21:09:40.733973+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:40.733973+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8448,"log_dups_size":0,"ondisk_log_size":8448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33617164,"num_objects":4104,"num_object_clones":0,"num_object_copies":16416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8446,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1567,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3287}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8312","reported_seq":8388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.471171+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:43.471171+0000","last_peered":"2024-09-15T21:09:43.471171+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:43.471171+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33749662,"num_objects":4121,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8311,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8358","reported_seq":8427,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.672885+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:40.672885+0000","last_peered":"2024-09-15T21:09:40.672885+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:40.672885+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8358,"log_dups_size":0,"ondisk_log_size":8358,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33568012,"num_objects":4098,"num_object_clones":0,"num_object_copies":16392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8356,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1764}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8442","reported_seq":8499,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.460040+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:41.460040+0000","last_peered":"2024-09-15T21:09:41.460040+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:41.460040+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8442,"log_dups_size":0,"ondisk_log_size":8442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34542860,"num_objects":4217,"num_object_clones":0,"num_object_copies":16868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1906}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8322","reported_seq":8367,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.419745+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:41.419745+0000","last_peered":"2024-09-15T21:09:41.419745+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:41.419745+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8322,"log_dups_size":0,"ondisk_log_size":8322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33210368,"num_objects":4054,"num_object_clones":0,"num_object_copies":16216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8322,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1738}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8556","reported_seq":8627,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.448221+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:40.448221+0000","last_peered":"2024-09-15T21:09:40.448221+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:40.448221+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8556,"log_dups_size":0,"ondisk_log_size":8556,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34534668,"num_objects":4216,"num_object_clones":0,"num_object_copies":16864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8554,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1850}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8619","reported_seq":8689,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.609759+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:40.609759+0000","last_peered":"2024-09-15T21:09:40.609759+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:40.609759+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8619,"log_dups_size":0,"ondisk_log_size":8619,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33928460,"num_objects":4142,"num_object_clones":0,"num_object_copies":16568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8617,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1668}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8384","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.418095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:41.418095+0000","last_peered":"2024-09-15T21:09:41.418095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:41.418095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8384,"log_dups_size":0,"ondisk_log_size":8384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33319448,"num_objects":4068,"num_object_clones":0,"num_object_copies":16272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8440","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.042002+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:43.042002+0000","last_peered":"2024-09-15T21:09:43.042002+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:43.042002+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764620,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8244","reported_seq":12908,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:41.397256+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:41.397256+0000","last_peered":"2024-09-15T21:09:41.397256+0000","last_clean":"2024-09-15T21:09:41.397256+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:41.397256+0000","last_undegraded":"2024-09-15T21:09:41.397256+0000","last_fullsized":"2024-09-15T21:09:41.397256+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8244,"log_dups_size":0,"ondisk_log_size":8244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33718272,"num_objects":4116,"num_object_clones":0,"num_object_copies":16464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8264","reported_seq":8304,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.293797+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:41.293797+0000","last_peered":"2024-09-15T21:09:41.293797+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:41.293797+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8264,"log_dups_size":0,"ondisk_log_size":8264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34101894,"num_objects":4163,"num_object_clones":0,"num_object_copies":16652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1907}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8362","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.337069+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:43.337069+0000","last_peered":"2024-09-15T21:09:43.337069+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:43.337069+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33905286,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8304","reported_seq":8360,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:41.352466+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:41.352466+0000","last_peered":"2024-09-15T21:09:41.352466+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:41.352466+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8304,"log_dups_size":0,"ondisk_log_size":8304,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33813772,"num_objects":4128,"num_object_clones":0,"num_object_copies":16512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8302,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1849}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8445","reported_seq":8528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.433624+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:43.433624+0000","last_peered":"2024-09-15T21:09:43.433624+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:43.433624+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34698508,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8391","reported_seq":8460,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:40.702445+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:40.702445+0000","last_peered":"2024-09-15T21:09:40.702445+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:40.702445+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8391,"log_dups_size":0,"ondisk_log_size":8391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34059532,"num_objects":4158,"num_object_clones":0,"num_object_copies":16632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8389,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":542070574,"num_objects":66176,"num_object_clones":0,"num_object_copies":264704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33258,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128236,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1589,"num_bytes_recovered":12804096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1072693248,"data_stored":1072693248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134149,"ondisk_log_size":134149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734468,"kb_used_data":265460,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93637372,"statfs":{"total":96636764160,"available":95884668928,"internally_reserved":0,"allocated":271831040,"data_stored":271666051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,6,4,0,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740916,"kb_used_data":264068,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93630924,"statfs":{"total":96636764160,"available":95878066176,"internally_reserved":0,"allocated":270405632,"data_stored":270240643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051653,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556612,"kb_used_data":254260,"kb_used_omap":4,"kb_used_meta":302331,"kb_avail":93815228,"statfs":{"total":96636764160,"available":96066793472,"internally_reserved":0,"allocated":260362240,"data_stored":260197251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":309587663},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726988,"kb_used_data":266364,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93644852,"statfs":{"total":96636764160,"available":95892328448,"internally_reserved":0,"allocated":272756736,"data_stored":272591747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271982592,"data_stored":271982592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":260079616,"data_stored":260079616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":271224832,"data_stored":271224832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:47.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:48.541 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:48.542 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:48.563 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-15T21:09:48.059878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534951726,"num_objects":65307,"num_object_clones":0,"num_object_copies":261228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129105,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135018,"ondisk_log_size":135018,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780192,"kb_used_data":1040804,"kb_used_omap":14,"kb_used_meta":1739249,"kb_avail":374707168,"statfs":{"total":386547056640,"available":383700140032,"internally_reserved":0,"allocated":1065783296,"data_stored":1065123340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1780991538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,18,14,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23560192,"num_objects":-2876,"num_object_clones":0,"num_object_copies":-11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003978"},"pg_stats":[{"pgid":"1.f","version":"33'8331","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.432502+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:46.432502+0000","last_peered":"2024-09-15T21:09:46.432502+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:46.432502+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8517","reported_seq":11810,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.733670+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:45.733670+0000","last_peered":"2024-09-15T21:09:45.733670+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:45.733670+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33051916,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8312","reported_seq":8388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.471171+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:43.471171+0000","last_peered":"2024-09-15T21:09:43.471171+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:43.471171+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33749662,"num_objects":4121,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8311,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8424","reported_seq":8494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.549400+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:45.549400+0000","last_peered":"2024-09-15T21:09:45.549400+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:45.549400+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027340,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8511","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.440997+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:46.440997+0000","last_peered":"2024-09-15T21:09:46.440997+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:46.440997+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33977612,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8394","reported_seq":8438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.325660+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:46.325660+0000","last_peered":"2024-09-15T21:09:46.325660+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:46.325660+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8394,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8620","reported_seq":8691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.621965+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:45.621965+0000","last_peered":"2024-09-15T21:09:45.621965+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:45.621965+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34010380,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8699","reported_seq":8769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.671639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:45.671639+0000","last_peered":"2024-09-15T21:09:45.671639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:45.671639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8452","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.336908+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:46.336908+0000","last_peered":"2024-09-15T21:09:46.336908+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:46.336908+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8452,"log_dups_size":0,"ondisk_log_size":8452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32762392,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8440","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.042002+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:43.042002+0000","last_peered":"2024-09-15T21:09:43.042002+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:43.042002+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764620,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8323","reported_seq":12987,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:46.378183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:46.378183+0000","last_peered":"2024-09-15T21:09:46.378183+0000","last_clean":"2024-09-15T21:09:46.378183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:46.378183+0000","last_undegraded":"2024-09-15T21:09:46.378183+0000","last_fullsized":"2024-09-15T21:09:46.378183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33071104,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8336","reported_seq":8376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.404902+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:46.404902+0000","last_peered":"2024-09-15T21:09:46.404902+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:46.404902+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512070,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8362","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.337069+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:43.337069+0000","last_peered":"2024-09-15T21:09:43.337069+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:43.337069+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33905286,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8383","reported_seq":8440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.509295+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:46.509295+0000","last_peered":"2024-09-15T21:09:46.509295+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:46.509295+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8383,"log_dups_size":0,"ondisk_log_size":8383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166604,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1770}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8445","reported_seq":8528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.433624+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:43.433624+0000","last_peered":"2024-09-15T21:09:43.433624+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:43.433624+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34698508,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8469","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.751034+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:45.751034+0000","last_peered":"2024-09-15T21:09:45.751034+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:45.751034+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420556,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534951726,"num_objects":65307,"num_object_clones":0,"num_object_copies":261228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129105,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1062924288,"data_stored":1062924288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135018,"ondisk_log_size":135018,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734120,"kb_used_data":260932,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93637720,"statfs":{"total":96636764160,"available":95885025280,"internally_reserved":0,"allocated":267194368,"data_stored":267029379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740916,"kb_used_data":264068,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93630924,"statfs":{"total":96636764160,"available":95878066176,"internally_reserved":0,"allocated":270405632,"data_stored":270240643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578572,"kb_used_data":254004,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93793268,"statfs":{"total":96636764160,"available":96044306432,"internally_reserved":0,"allocated":260100096,"data_stored":259935107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726584,"kb_used_data":261800,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93645256,"statfs":{"total":96636764160,"available":95892742144,"internally_reserved":0,"allocated":268083200,"data_stored":267918211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267243520,"data_stored":267243520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259809280,"data_stored":259809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266465280,"data_stored":266465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:48.564 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:49.244 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:49.245 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:49.277 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-15T21:09:48.059878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":534951726,"num_objects":65307,"num_object_clones":0,"num_object_copies":261228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129105,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135018,"ondisk_log_size":135018,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2780192,"kb_used_data":1040804,"kb_used_omap":14,"kb_used_meta":1739249,"kb_avail":374707168,"statfs":{"total":386547056640,"available":383700140032,"internally_reserved":0,"allocated":1065783296,"data_stored":1065123340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1780991538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,18,14,14,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-23560192,"num_objects":-2876,"num_object_clones":0,"num_object_copies":-11504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2876,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003978"},"pg_stats":[{"pgid":"1.f","version":"33'8331","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.432502+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:46.432502+0000","last_peered":"2024-09-15T21:09:46.432502+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:46.432502+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8517","reported_seq":11810,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.733670+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:45.733670+0000","last_peered":"2024-09-15T21:09:45.733670+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:45.733670+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33051916,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8312","reported_seq":8388,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.471171+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:43.471171+0000","last_peered":"2024-09-15T21:09:43.471171+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:43.471171+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8312,"log_dups_size":0,"ondisk_log_size":8312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33749662,"num_objects":4121,"num_object_clones":0,"num_object_copies":16484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4121,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8311,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1863}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8424","reported_seq":8494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.549400+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:45.549400+0000","last_peered":"2024-09-15T21:09:45.549400+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:45.549400+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027340,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8511","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.440997+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:46.440997+0000","last_peered":"2024-09-15T21:09:46.440997+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:46.440997+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33977612,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8394","reported_seq":8438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.325660+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:46.325660+0000","last_peered":"2024-09-15T21:09:46.325660+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:46.325660+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8394,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8620","reported_seq":8691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.621965+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:45.621965+0000","last_peered":"2024-09-15T21:09:45.621965+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:45.621965+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34010380,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8699","reported_seq":8769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.671639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:45.671639+0000","last_peered":"2024-09-15T21:09:45.671639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:45.671639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8452","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.336908+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:46.336908+0000","last_peered":"2024-09-15T21:09:46.336908+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:46.336908+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8452,"log_dups_size":0,"ondisk_log_size":8452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32762392,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8440","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.042002+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:43.042002+0000","last_peered":"2024-09-15T21:09:43.042002+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:43.042002+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8440,"log_dups_size":0,"ondisk_log_size":8440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33764620,"num_objects":4122,"num_object_clones":0,"num_object_copies":16488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8438,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1787}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8323","reported_seq":12987,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:46.378183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:46.378183+0000","last_peered":"2024-09-15T21:09:46.378183+0000","last_clean":"2024-09-15T21:09:46.378183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:46.378183+0000","last_undegraded":"2024-09-15T21:09:46.378183+0000","last_fullsized":"2024-09-15T21:09:46.378183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33071104,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8336","reported_seq":8376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.404902+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:46.404902+0000","last_peered":"2024-09-15T21:09:46.404902+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:46.404902+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512070,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8362","reported_seq":8443,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.337069+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:43.337069+0000","last_peered":"2024-09-15T21:09:43.337069+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:43.337069+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8362,"log_dups_size":0,"ondisk_log_size":8362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33905286,"num_objects":4139,"num_object_clones":0,"num_object_copies":16556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8361,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8383","reported_seq":8440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.509295+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:46.509295+0000","last_peered":"2024-09-15T21:09:46.509295+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:46.509295+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8383,"log_dups_size":0,"ondisk_log_size":8383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166604,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1770}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8445","reported_seq":8528,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:43.433624+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:43.433624+0000","last_peered":"2024-09-15T21:09:43.433624+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:43.433624+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8445,"log_dups_size":0,"ondisk_log_size":8445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34698508,"num_objects":4236,"num_object_clones":0,"num_object_copies":16944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8443,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1955}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8469","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.751034+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:45.751034+0000","last_peered":"2024-09-15T21:09:45.751034+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:45.751034+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420556,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":534951726,"num_objects":65307,"num_object_clones":0,"num_object_copies":261228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129105,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1062924288,"data_stored":1062924288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135018,"ondisk_log_size":135018,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734120,"kb_used_data":260932,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93637720,"statfs":{"total":96636764160,"available":95885025280,"internally_reserved":0,"allocated":267194368,"data_stored":267029379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740916,"kb_used_data":264068,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":93630924,"statfs":{"total":96636764160,"available":95878066176,"internally_reserved":0,"allocated":270405632,"data_stored":270240643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,6,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051654,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":578572,"kb_used_data":254004,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93793268,"statfs":{"total":96636764160,"available":96044306432,"internally_reserved":0,"allocated":260100096,"data_stored":259935107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,2,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726584,"kb_used_data":261800,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93645256,"statfs":{"total":96636764160,"available":95892742144,"internally_reserved":0,"allocated":268083200,"data_stored":267918211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267243520,"data_stored":267243520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259809280,"data_stored":259809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":269406208,"data_stored":269406208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266465280,"data_stored":266465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:49.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:49.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8331", 'reported_seq': 8377, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.432502+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:46.432502+0000', 'last_peered': '2024-09-15T21:09:46.432502+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:46.432502+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8331, 'log_dups_size': 0, 'ondisk_log_size': 8331, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32940032, 'num_objects': 4021, 'num_object_clones': 0, 'num_object_copies': 16084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8331, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1745}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8517", 'reported_seq': 11810, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:45.733670+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:45.733670+0000', 'last_peered': '2024-09-15T21:09:45.733670+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:45.733670+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8517, 'log_dups_size': 0, 'ondisk_log_size': 8517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33051916, 'num_objects': 4035, 'num_object_clones': 0, 'num_object_copies': 16140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 770, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8515, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1614, 'num_bytes_recovered': 13189120, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8312", 'reported_seq': 8388, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:43.471171+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:43.471171+0000', 'last_peered': '2024-09-15T21:09:43.471171+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:43.471171+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8312, 'log_dups_size': 0, 'ondisk_log_size': 8312, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33749662, 'num_objects': 4121, 'num_object_clones': 0, 'num_object_copies': 16484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4121, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8311, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1863}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8424", 'reported_seq': 8494, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:45.549400+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:45.549400+0000', 'last_peered': '2024-09-15T21:09:45.549400+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:45.549400+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8424, 'log_dups_size': 0, 'ondisk_log_size': 8424, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33027340, 'num_objects': 4032, 'num_object_clones': 0, 'num_object_copies': 16128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4032, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8422, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1698}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8511", 'reported_seq': 8568, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.440997+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:46.440997+0000', 'last_peered': '2024-09-15T21:09:46.440997+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:46.440997+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8511, 'log_dups_size': 0, 'ondisk_log_size': 8511, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33977612, 'num_objects': 4148, 'num_object_clones': 0, 'num_object_copies': 16592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4148, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8509, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1837}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8394", 'reported_seq': 8438, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.325660+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:46.325660+0000', 'last_peered': '2024-09-15T21:09:46.325660+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:46.325660+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8394, 'log_dups_size': 0, 'ondisk_log_size': 8394, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32620544, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 15928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8394, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1666}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:49.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8620", 'reported_seq': 8691, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:45.621965+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:45.621965+0000', 'last_peered': '2024-09-15T21:09:45.621965+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:45.621965+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8620, 'log_dups_size': 0, 'ondisk_log_size': 8620, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34010380, 'num_objects': 4152, 'num_object_clones': 0, 'num_object_copies': 16608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8618, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1786}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8699", 'reported_seq': 8769, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:45.671639+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:45.671639+0000', 'last_peered': '2024-09-15T21:09:45.671639+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:45.671639+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8699, 'log_dups_size': 0, 'ondisk_log_size': 8699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33273100, 'num_objects': 4062, 'num_object_clones': 0, 'num_object_copies': 16248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8697, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1588}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8452", 'reported_seq': 8513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.336908+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:46.336908+0000', 'last_peered': '2024-09-15T21:09:46.336908+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:46.336908+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8452, 'log_dups_size': 0, 'ondisk_log_size': 8452, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32762392, 'num_objects': 4000, 'num_object_clones': 0, 'num_object_copies': 16000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4000, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8449, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1668}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8440", 'reported_seq': 8522, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:43.042002+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:43.042002+0000', 'last_peered': '2024-09-15T21:09:43.042002+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:43.042002+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8440, 'log_dups_size': 0, 'ondisk_log_size': 8440, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33764620, 'num_objects': 4122, 'num_object_clones': 0, 'num_object_copies': 16488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8438, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1787}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:49.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8336", 'reported_seq': 8376, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.404902+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:46.404902+0000', 'last_peered': '2024-09-15T21:09:46.404902+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:46.404902+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8336, 'log_dups_size': 0, 'ondisk_log_size': 8336, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33512070, 'num_objects': 4091, 'num_object_clones': 0, 'num_object_copies': 16364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4091, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8335, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1835}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8362", 'reported_seq': 8443, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:43.337069+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:43.337069+0000', 'last_peered': '2024-09-15T21:09:43.337069+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:43.337069+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8362, 'log_dups_size': 0, 'ondisk_log_size': 8362, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33905286, 'num_objects': 4139, 'num_object_clones': 0, 'num_object_copies': 16556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8361, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1853}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8383", 'reported_seq': 8440, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:46.509295+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:46.509295+0000', 'last_peered': '2024-09-15T21:09:46.509295+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:46.509295+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8383, 'log_dups_size': 0, 'ondisk_log_size': 8383, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33166604, 'num_objects': 4049, 'num_object_clones': 0, 'num_object_copies': 16196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8381, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8445", 'reported_seq': 8528, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:43.433624+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:43.433624+0000', 'last_peered': '2024-09-15T21:09:43.433624+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:43.433624+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8445, 'log_dups_size': 0, 'ondisk_log_size': 8445, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 34698508, 'num_objects': 4236, 'num_object_clones': 0, 'num_object_copies': 16944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8443, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1955}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:49.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8469", 'reported_seq': 8538, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:45.751034+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:45.751034+0000', 'last_peered': '2024-09-15T21:09:45.751034+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:45.751034+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8469, 'log_dups_size': 0, 'ondisk_log_size': 8469, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33420556, 'num_objects': 4080, 'num_object_clones': 0, 'num_object_copies': 16320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8467, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:49.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:50.134 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:50.134 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33211/261228 objects degraded (12.713%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":455,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":65307,"data_bytes":534951726,"bytes_used":2846916608,"bytes_avail":383700140032,"bytes_total":386547056640,"degraded_objects":33211,"degraded_total":261228,"degraded_ratio":0.12713415100984579,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":68244,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":239},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:50.161 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:50.162 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:50.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:50.795 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:50.819 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-09-15T21:09:50.060590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532428590,"num_objects":64999,"num_object_clones":0,"num_object_copies":259996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129413,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135326,"ondisk_log_size":135326,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766964,"kb_used_data":1035932,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374720396,"statfs":{"total":386547056640,"available":383713685504,"internally_reserved":0,"allocated":1060794368,"data_stored":1060134412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1772471858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,1,0,2,10,14,12,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21766144,"num_objects":-2657,"num_object_clones":0,"num_object_copies":-10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2657,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004021"},"pg_stats":[{"pgid":"1.f","version":"33'8331","reported_seq":8377,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.432502+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:46.432502+0000","last_peered":"2024-09-15T21:09:46.432502+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:46.432502+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8331,"log_dups_size":0,"ondisk_log_size":8331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32940032,"num_objects":4021,"num_object_clones":0,"num_object_copies":16084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8331,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8517","reported_seq":11810,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.733670+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:45.733670+0000","last_peered":"2024-09-15T21:09:45.733670+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:45.733670+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33051916,"num_objects":4035,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8515,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8382","reported_seq":8463,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:48.321820+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:48.321820+0000","last_peered":"2024-09-15T21:09:48.321820+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:48.321820+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8382,"log_dups_size":0,"ondisk_log_size":8382,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33176222,"num_objects":4051,"num_object_clones":0,"num_object_copies":16204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8381,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1793}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8424","reported_seq":8494,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.549400+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:45.549400+0000","last_peered":"2024-09-15T21:09:45.549400+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:45.549400+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8424,"log_dups_size":0,"ondisk_log_size":8424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33027340,"num_objects":4032,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8422,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8511","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.440997+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:46.440997+0000","last_peered":"2024-09-15T21:09:46.440997+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:46.440997+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33977612,"num_objects":4148,"num_object_clones":0,"num_object_copies":16592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8509,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1837}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8394","reported_seq":8438,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.325660+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:46.325660+0000","last_peered":"2024-09-15T21:09:46.325660+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:46.325660+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8394,"log_dups_size":0,"ondisk_log_size":8394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32620544,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8394,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1666}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8620","reported_seq":8691,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.621965+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:45.621965+0000","last_peered":"2024-09-15T21:09:45.621965+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:45.621965+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8620,"log_dups_size":0,"ondisk_log_size":8620,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34010380,"num_objects":4152,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8618,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1786}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8699","reported_seq":8769,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.671639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:45.671639+0000","last_peered":"2024-09-15T21:09:45.671639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:45.671639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33273100,"num_objects":4062,"num_object_clones":0,"num_object_copies":16248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8697,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1588}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8452","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.336908+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:46.336908+0000","last_peered":"2024-09-15T21:09:46.336908+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:46.336908+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8452,"log_dups_size":0,"ondisk_log_size":8452,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32762392,"num_objects":4000,"num_object_clones":0,"num_object_copies":16000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8449,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8522","reported_seq":8604,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:48.250024+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:48.250024+0000","last_peered":"2024-09-15T21:09:48.250024+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:48.250024+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8522,"log_dups_size":0,"ondisk_log_size":8522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33092876,"num_objects":4040,"num_object_clones":0,"num_object_copies":16160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1705}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8323","reported_seq":12987,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:46.378183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:46.378183+0000","last_peered":"2024-09-15T21:09:46.378183+0000","last_clean":"2024-09-15T21:09:46.378183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:46.378183+0000","last_undegraded":"2024-09-15T21:09:46.378183+0000","last_fullsized":"2024-09-15T21:09:46.378183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33071104,"num_objects":4037,"num_object_clones":0,"num_object_copies":16148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8336","reported_seq":8376,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.404902+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:46.404902+0000","last_peered":"2024-09-15T21:09:46.404902+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:46.404902+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8336,"log_dups_size":0,"ondisk_log_size":8336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33512070,"num_objects":4091,"num_object_clones":0,"num_object_copies":16364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8335,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1835}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8441","reported_seq":8522,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:48.373049+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:48.373049+0000","last_peered":"2024-09-15T21:09:48.373049+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:48.373049+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8441,"log_dups_size":0,"ondisk_log_size":8441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33258118,"num_objects":4060,"num_object_clones":0,"num_object_copies":16240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8440,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1774}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8383","reported_seq":8440,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:46.509295+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:46.509295+0000","last_peered":"2024-09-15T21:09:46.509295+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:46.509295+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8383,"log_dups_size":0,"ondisk_log_size":8383,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33166604,"num_objects":4049,"num_object_clones":0,"num_object_copies":16196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1770}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8522","reported_seq":8606,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:48.403365+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:48.403365+0000","last_peered":"2024-09-15T21:09:48.403365+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:48.403365+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8522,"log_dups_size":0,"ondisk_log_size":8522,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34067724,"num_objects":4159,"num_object_clones":0,"num_object_copies":16636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8520,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1878}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8469","reported_seq":8538,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:45.751034+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:45.751034+0000","last_peered":"2024-09-15T21:09:45.751034+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:45.751034+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8469,"log_dups_size":0,"ondisk_log_size":8469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33420556,"num_objects":4080,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1765}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":532428590,"num_objects":64999,"num_object_clones":0,"num_object_copies":259996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64999,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129413,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1636,"num_bytes_recovered":13189120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1057746944,"data_stored":1057746944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135326,"ondisk_log_size":135326,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734120,"kb_used_data":260932,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93637720,"statfs":{"total":96636764160,"available":95885025280,"internally_reserved":0,"allocated":267194368,"data_stored":267029379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,5,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723772,"kb_used_data":259432,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93648068,"statfs":{"total":96636764160,"available":95895621632,"internally_reserved":0,"allocated":265658368,"data_stored":265493379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582488,"kb_used_data":253768,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93789352,"statfs":{"total":96636764160,"available":96040296448,"internally_reserved":0,"allocated":259858432,"data_stored":259693443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247892,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726584,"kb_used_data":261800,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93645256,"statfs":{"total":96636764160,"available":95892742144,"internally_reserved":0,"allocated":268083200,"data_stored":267918211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":267243520,"data_stored":267243520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259530752,"data_stored":259530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":264507392,"data_stored":264507392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266465280,"data_stored":266465280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:53.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:54.442 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:54.443 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:54.464 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-15T21:09:54.062316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523220782,"num_objects":63875,"num_object_clones":0,"num_object_copies":255500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130537,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1689,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136450,"ondisk_log_size":136450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749112,"kb_used_data":1022264,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374738248,"statfs":{"total":386547056640,"available":383731965952,"internally_reserved":0,"allocated":1046798336,"data_stored":1046138380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1768212018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,2,7,14,12,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21544960,"num_objects":-2630,"num_object_clones":0,"num_object_copies":-10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004157"},"pg_stats":[{"pgid":"1.f","version":"33'8399","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.270562+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:51.270562+0000","last_peered":"2024-09-15T21:09:51.270562+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:51.270562+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8399,"log_dups_size":0,"ondisk_log_size":8399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8585","reported_seq":11984,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.813248+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:50.813248+0000","last_peered":"2024-09-15T21:09:50.813248+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:50.813248+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32494860,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8450","reported_seq":8531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.363733+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:53.363733+0000","last_peered":"2024-09-15T21:09:53.363733+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:53.363733+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32619166,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8449,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8499","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.714169+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:50.714169+0000","last_peered":"2024-09-15T21:09:50.714169+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:50.714169+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8499,"log_dups_size":0,"ondisk_log_size":8499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32412940,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8497,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1623}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8584","reported_seq":8640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.297083+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:51.297083+0000","last_peered":"2024-09-15T21:09:51.297083+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:51.297083+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379596,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8457","reported_seq":8502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.536157+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:51.536157+0000","last_peered":"2024-09-15T21:09:51.536157+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:51.536157+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32104448,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8695","reported_seq":8766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.753322+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:50.753322+0000","last_peered":"2024-09-15T21:09:50.753322+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:50.753322+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33395980,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1711}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8769","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.546144+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:50.546144+0000","last_peered":"2024-09-15T21:09:50.546144+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:50.546144+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8517","reported_seq":8577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.527986+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:51.527986+0000","last_peered":"2024-09-15T21:09:51.527986+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:51.527986+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32229912,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8595","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.473417+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:53.473417+0000","last_peered":"2024-09-15T21:09:53.473417+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:53.473417+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8595,"log_dups_size":0,"ondisk_log_size":8595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32494860,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1632}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8386","reported_seq":13050,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:51.171216+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:51.171216+0000","last_peered":"2024-09-15T21:09:51.171216+0000","last_clean":"2024-09-15T21:09:51.171216+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:51.171216+0000","last_undegraded":"2024-09-15T21:09:51.171216+0000","last_fullsized":"2024-09-15T21:09:51.171216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8420","reported_seq":8459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.327105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:51.327105+0000","last_peered":"2024-09-15T21:09:51.327105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:51.327105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32823942,"num_objects":4007,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8517","reported_seq":8598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.379007+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:53.379007+0000","last_peered":"2024-09-15T21:09:53.379007+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:53.379007+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32635526,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8450","reported_seq":8506,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.499056+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:51.499056+0000","last_peered":"2024-09-15T21:09:51.499056+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:51.499056+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32617740,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8597","reported_seq":8681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.475199+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:53.475199+0000","last_peered":"2024-09-15T21:09:53.475199+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:53.475199+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33453324,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8530","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.821417+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:50.821417+0000","last_peered":"2024-09-15T21:09:50.821417+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:50.821417+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32920844,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523220782,"num_objects":63875,"num_object_clones":0,"num_object_copies":255500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130537,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1689,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1043668992,"data_stored":1043668992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136450,"ondisk_log_size":136450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717120,"kb_used_data":256432,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93654720,"statfs":{"total":96636764160,"available":95902433280,"internally_reserved":0,"allocated":262586368,"data_stored":262421379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723392,"kb_used_data":254896,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93648448,"statfs":{"total":96636764160,"available":95896010752,"internally_reserved":0,"allocated":261013504,"data_stored":260848515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582488,"kb_used_data":253768,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93789352,"statfs":{"total":96636764160,"available":96040296448,"internally_reserved":0,"allocated":259858432,"data_stored":259693443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726112,"kb_used_data":257168,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93645728,"statfs":{"total":96636764160,"available":95893225472,"internally_reserved":0,"allocated":263340032,"data_stored":263175043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262504448,"data_stored":262504448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259530752,"data_stored":259530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259825664,"data_stored":259825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261808128,"data_stored":261808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:54.465 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:55.096 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:55.096 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:55.123 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-09-15T21:09:54.062316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":523220782,"num_objects":63875,"num_object_clones":0,"num_object_copies":255500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130537,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1689,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136450,"ondisk_log_size":136450,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749112,"kb_used_data":1022264,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374738248,"statfs":{"total":386547056640,"available":383731965952,"internally_reserved":0,"allocated":1046798336,"data_stored":1046138380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1768212018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,2,7,14,12,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21544960,"num_objects":-2630,"num_object_clones":0,"num_object_copies":-10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-100,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2630,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004157"},"pg_stats":[{"pgid":"1.f","version":"33'8399","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.270562+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:51.270562+0000","last_peered":"2024-09-15T21:09:51.270562+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:51.270562+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8399,"log_dups_size":0,"ondisk_log_size":8399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8585","reported_seq":11984,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.813248+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:50.813248+0000","last_peered":"2024-09-15T21:09:50.813248+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:50.813248+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32494860,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3250}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8450","reported_seq":8531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.363733+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:53.363733+0000","last_peered":"2024-09-15T21:09:53.363733+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:53.363733+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32619166,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8449,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8499","reported_seq":8568,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.714169+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:50.714169+0000","last_peered":"2024-09-15T21:09:50.714169+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:50.714169+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8499,"log_dups_size":0,"ondisk_log_size":8499,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32412940,"num_objects":3957,"num_object_clones":0,"num_object_copies":15828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8497,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1623}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8584","reported_seq":8640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.297083+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:51.297083+0000","last_peered":"2024-09-15T21:09:51.297083+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:51.297083+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379596,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8457","reported_seq":8502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.536157+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:51.536157+0000","last_peered":"2024-09-15T21:09:51.536157+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:51.536157+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32104448,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8695","reported_seq":8766,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.753322+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:50.753322+0000","last_peered":"2024-09-15T21:09:50.753322+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:50.753322+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8695,"log_dups_size":0,"ondisk_log_size":8695,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33395980,"num_objects":4077,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8693,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1711}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8769","reported_seq":8839,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.546144+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:50.546144+0000","last_peered":"2024-09-15T21:09:50.546144+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:50.546144+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8769,"log_dups_size":0,"ondisk_log_size":8769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32699660,"num_objects":3992,"num_object_clones":0,"num_object_copies":15968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8767,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1518}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8517","reported_seq":8577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.527986+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:51.527986+0000","last_peered":"2024-09-15T21:09:51.527986+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:51.527986+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32229912,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8595","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.473417+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:53.473417+0000","last_peered":"2024-09-15T21:09:53.473417+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:53.473417+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8595,"log_dups_size":0,"ondisk_log_size":8595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32494860,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1632}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8386","reported_seq":13050,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:51.171216+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:51.171216+0000","last_peered":"2024-09-15T21:09:51.171216+0000","last_clean":"2024-09-15T21:09:51.171216+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:51.171216+0000","last_undegraded":"2024-09-15T21:09:51.171216+0000","last_fullsized":"2024-09-15T21:09:51.171216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8420","reported_seq":8459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.327105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:51.327105+0000","last_peered":"2024-09-15T21:09:51.327105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:51.327105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32823942,"num_objects":4007,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8517","reported_seq":8598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.379007+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:53.379007+0000","last_peered":"2024-09-15T21:09:53.379007+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:53.379007+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32635526,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8450","reported_seq":8506,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.499056+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:51.499056+0000","last_peered":"2024-09-15T21:09:51.499056+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:51.499056+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32617740,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8597","reported_seq":8681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.475199+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:53.475199+0000","last_peered":"2024-09-15T21:09:53.475199+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:53.475199+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33453324,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8530","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:50.821417+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:50.821417+0000","last_peered":"2024-09-15T21:09:50.821417+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:50.821417+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8530,"log_dups_size":0,"ondisk_log_size":8530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32920844,"num_objects":4019,"num_object_clones":0,"num_object_copies":16076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8528,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1704}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":523220782,"num_objects":63875,"num_object_clones":0,"num_object_copies":255500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33158,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130537,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1689,"num_bytes_recovered":13623296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1043668992,"data_stored":1043668992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136450,"ondisk_log_size":136450,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717120,"kb_used_data":256432,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93654720,"statfs":{"total":96636764160,"available":95902433280,"internally_reserved":0,"allocated":262586368,"data_stored":262421379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723392,"kb_used_data":254896,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93648448,"statfs":{"total":96636764160,"available":95896010752,"internally_reserved":0,"allocated":261013504,"data_stored":260848515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051655,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582488,"kb_used_data":253768,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93789352,"statfs":{"total":96636764160,"available":96040296448,"internally_reserved":0,"allocated":259858432,"data_stored":259693443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,0,3,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247893,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726112,"kb_used_data":257168,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93645728,"statfs":{"total":96636764160,"available":95893225472,"internally_reserved":0,"allocated":263340032,"data_stored":263175043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,6,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262504448,"data_stored":262504448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259530752,"data_stored":259530752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259825664,"data_stored":259825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261808128,"data_stored":261808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8399", 'reported_seq': 8445, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.270562+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:51.270562+0000', 'last_peered': '2024-09-15T21:09:51.270562+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:51.270562+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8399, 'log_dups_size': 0, 'ondisk_log_size': 8399, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32382976, 'num_objects': 3953, 'num_object_clones': 0, 'num_object_copies': 15812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3953, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8399, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1677}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8585", 'reported_seq': 11984, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:50.813248+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:50.813248+0000', 'last_peered': '2024-09-15T21:09:50.813248+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:50.813248+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8585, 'log_dups_size': 0, 'ondisk_log_size': 8585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32494860, 'num_objects': 3967, 'num_object_clones': 0, 'num_object_copies': 15868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 717, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8583, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1667, 'num_bytes_recovered': 13623296, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3250}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8450", 'reported_seq': 8531, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:53.363733+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:53.363733+0000', 'last_peered': '2024-09-15T21:09:53.363733+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:53.363733+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8450, 'log_dups_size': 0, 'ondisk_log_size': 8450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32619166, 'num_objects': 3983, 'num_object_clones': 0, 'num_object_copies': 15932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3983, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8449, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1725}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:09:55.124 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8499", 'reported_seq': 8568, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:50.714169+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:50.714169+0000', 'last_peered': '2024-09-15T21:09:50.714169+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:50.714169+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8499, 'log_dups_size': 0, 'ondisk_log_size': 8499, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32412940, 'num_objects': 3957, 'num_object_clones': 0, 'num_object_copies': 15828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8497, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1623}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8584", 'reported_seq': 8640, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.297083+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:51.297083+0000', 'last_peered': '2024-09-15T21:09:51.297083+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:51.297083+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8584, 'log_dups_size': 0, 'ondisk_log_size': 8584, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33379596, 'num_objects': 4075, 'num_object_clones': 0, 'num_object_copies': 16300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8582, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1764}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8457", 'reported_seq': 8502, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.536157+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:51.536157+0000', 'last_peered': '2024-09-15T21:09:51.536157+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:51.536157+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8457, 'log_dups_size': 0, 'ondisk_log_size': 8457, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32104448, 'num_objects': 3919, 'num_object_clones': 0, 'num_object_copies': 15676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8457, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8695", 'reported_seq': 8766, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:50.753322+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:50.753322+0000', 'last_peered': '2024-09-15T21:09:50.753322+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:50.753322+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8695, 'log_dups_size': 0, 'ondisk_log_size': 8695, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33395980, 'num_objects': 4077, 'num_object_clones': 0, 'num_object_copies': 16308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4077, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8693, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1711}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8769", 'reported_seq': 8839, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:50.546144+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:50.546144+0000', 'last_peered': '2024-09-15T21:09:50.546144+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:50.546144+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8769, 'log_dups_size': 0, 'ondisk_log_size': 8769, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32699660, 'num_objects': 3992, 'num_object_clones': 0, 'num_object_copies': 15968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3992, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8767, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1518}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8517", 'reported_seq': 8577, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.527986+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:51.527986+0000', 'last_peered': '2024-09-15T21:09:51.527986+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:51.527986+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8517, 'log_dups_size': 0, 'ondisk_log_size': 8517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32229912, 'num_objects': 3935, 'num_object_clones': 0, 'num_object_copies': 15740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3935, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8514, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1603}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8595", 'reported_seq': 8676, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:53.473417+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:53.473417+0000', 'last_peered': '2024-09-15T21:09:53.473417+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:53.473417+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8595, 'log_dups_size': 0, 'ondisk_log_size': 8595, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32494860, 'num_objects': 3967, 'num_object_clones': 0, 'num_object_copies': 15868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3967, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8593, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1632}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8420", 'reported_seq': 8459, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.327105+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:51.327105+0000', 'last_peered': '2024-09-15T21:09:51.327105+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:51.327105+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8420, 'log_dups_size': 0, 'ondisk_log_size': 8420, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32823942, 'num_objects': 4007, 'num_object_clones': 0, 'num_object_copies': 16028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8419, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1751}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8517", 'reported_seq': 8598, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:53.379007+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:53.379007+0000', 'last_peered': '2024-09-15T21:09:53.379007+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:53.379007+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8517, 'log_dups_size': 0, 'ondisk_log_size': 8517, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32635526, 'num_objects': 3984, 'num_object_clones': 0, 'num_object_copies': 15936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8516, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1698}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8450", 'reported_seq': 8506, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:51.499056+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:51.499056+0000', 'last_peered': '2024-09-15T21:09:51.499056+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:51.499056+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8450, 'log_dups_size': 0, 'ondisk_log_size': 8450, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32617740, 'num_objects': 3982, 'num_object_clones': 0, 'num_object_copies': 15928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3982, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8448, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1703}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8597", 'reported_seq': 8681, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:53.475199+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:53.475199+0000', 'last_peered': '2024-09-15T21:09:53.475199+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:53.475199+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8597, 'log_dups_size': 0, 'ondisk_log_size': 8597, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 33453324, 'num_objects': 4084, 'num_object_clones': 0, 'num_object_copies': 16336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4084, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8595, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1803}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:09:55.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8530", 'reported_seq': 8600, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:50.821417+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:50.821417+0000', 'last_peered': '2024-09-15T21:09:50.821417+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:50.821417+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8530, 'log_dups_size': 0, 'ondisk_log_size': 8530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32920844, 'num_objects': 4019, 'num_object_clones': 0, 'num_object_copies': 16076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4019, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8528, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:09:55.127 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:09:55.903 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:55.903 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33158/255500 objects degraded (12.978%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":461,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":63875,"data_bytes":523220782,"bytes_used":2815090688,"bytes_avail":383731965952,"bytes_total":386547056640,"degraded_objects":33158,"degraded_total":255500,"degraded_ratio":0.12977690802348338,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":68243,"recovering_keys_per_sec":0,"num_objects_recovered":100,"num_bytes_recovered":819200,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":219},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:09:55.935 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:09:55.936 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:09:56.565 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:09:56.565 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:09:56.592 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-15T21:09:56.063103+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":519968558,"num_objects":63478,"num_object_clones":0,"num_object_copies":253912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130934,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136847,"ondisk_log_size":136847,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735860,"kb_used_data":1017332,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374751500,"statfs":{"total":386547056640,"available":383745536000,"internally_reserved":0,"allocated":1041747968,"data_stored":1041088012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1759692338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,3,9,5,8,10,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22102016,"num_objects":-2698,"num_object_clones":0,"num_object_copies":-10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":1146880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004396"},"pg_stats":[{"pgid":"1.f","version":"33'8399","reported_seq":8445,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.270562+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:51.270562+0000","last_peered":"2024-09-15T21:09:51.270562+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:51.270562+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8399,"log_dups_size":0,"ondisk_log_size":8399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32382976,"num_objects":3953,"num_object_clones":0,"num_object_copies":15812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1677}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8671","reported_seq":12163,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.860886+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:55.860886+0000","last_peered":"2024-09-15T21:09:55.860886+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:55.860886+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31790348,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8450","reported_seq":8531,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.363733+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:53.363733+0000","last_peered":"2024-09-15T21:09:53.363733+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:53.363733+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32619166,"num_objects":3983,"num_object_clones":0,"num_object_copies":15932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3983,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8449,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1725}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8572","reported_seq":8642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.772583+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:55.772583+0000","last_peered":"2024-09-15T21:09:55.772583+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:55.772583+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31814924,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8584","reported_seq":8640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.297083+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:51.297083+0000","last_peered":"2024-09-15T21:09:51.297083+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:51.297083+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8584,"log_dups_size":0,"ondisk_log_size":8584,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33379596,"num_objects":4075,"num_object_clones":0,"num_object_copies":16300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1764}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8457","reported_seq":8502,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.536157+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:51.536157+0000","last_peered":"2024-09-15T21:09:51.536157+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:51.536157+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8457,"log_dups_size":0,"ondisk_log_size":8457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32104448,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8764","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908625+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:55.908625+0000","last_peered":"2024-09-15T21:09:55.908625+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:55.908625+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830732,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8866","reported_seq":8936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.689373+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:55.689373+0000","last_peered":"2024-09-15T21:09:55.689373+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:55.689373+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31905036,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8517","reported_seq":8577,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.527986+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:51.527986+0000","last_peered":"2024-09-15T21:09:51.527986+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:51.527986+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32229912,"num_objects":3935,"num_object_clones":0,"num_object_copies":15740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8514,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1603}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8595","reported_seq":8676,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.473417+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:53.473417+0000","last_peered":"2024-09-15T21:09:53.473417+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:53.473417+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8595,"log_dups_size":0,"ondisk_log_size":8595,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32494860,"num_objects":3967,"num_object_clones":0,"num_object_copies":15868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1632}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8386","reported_seq":13050,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:51.171216+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:51.171216+0000","last_peered":"2024-09-15T21:09:51.171216+0000","last_clean":"2024-09-15T21:09:51.171216+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:51.171216+0000","last_undegraded":"2024-09-15T21:09:51.171216+0000","last_fullsized":"2024-09-15T21:09:51.171216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32555008,"num_objects":3974,"num_object_clones":0,"num_object_copies":15896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2497,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8420","reported_seq":8459,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.327105+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:51.327105+0000","last_peered":"2024-09-15T21:09:51.327105+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:51.327105+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32823942,"num_objects":4007,"num_object_clones":0,"num_object_copies":16028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8419,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1751}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8517","reported_seq":8598,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.379007+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:53.379007+0000","last_peered":"2024-09-15T21:09:53.379007+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:53.379007+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8517,"log_dups_size":0,"ondisk_log_size":8517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32635526,"num_objects":3984,"num_object_clones":0,"num_object_copies":15936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1698}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8450","reported_seq":8506,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:51.499056+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:51.499056+0000","last_peered":"2024-09-15T21:09:51.499056+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:51.499056+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8450,"log_dups_size":0,"ondisk_log_size":8450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32617740,"num_objects":3982,"num_object_clones":0,"num_object_copies":15928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1703}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8597","reported_seq":8681,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:53.475199+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:53.475199+0000","last_peered":"2024-09-15T21:09:53.475199+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:53.475199+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8597,"log_dups_size":0,"ondisk_log_size":8597,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33453324,"num_objects":4084,"num_object_clones":0,"num_object_copies":16336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8595,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1803}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8602","reported_seq":8672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908286+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:55.908286+0000","last_peered":"2024-09-15T21:09:55.908286+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:55.908286+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32331020,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":519968558,"num_objects":63478,"num_object_clones":0,"num_object_copies":253912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130934,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1038602240,"data_stored":1038602240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136847,"ondisk_log_size":136847,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717120,"kb_used_data":256432,"kb_used_omap":4,"kb_used_meta":460667,"kb_avail":93654720,"statfs":{"total":96636764160,"available":95902433280,"internally_reserved":0,"allocated":262586368,"data_stored":262421379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":471723728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723392,"kb_used_data":254896,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93648448,"statfs":{"total":96636764160,"available":95896010752,"internally_reserved":0,"allocated":261013504,"data_stored":260848515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051656,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":569668,"kb_used_data":253428,"kb_used_omap":4,"kb_used_meta":316219,"kb_avail":93802172,"statfs":{"total":96636764160,"available":96053424128,"internally_reserved":0,"allocated":259510272,"data_stored":259345283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":323808975},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725680,"kb_used_data":252576,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93646160,"statfs":{"total":96636764160,"available":95893667840,"internally_reserved":0,"allocated":258637824,"data_stored":258472835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,5,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257732608,"data_stored":257732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":259235840,"data_stored":259235840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":259825664,"data_stored":259825664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":261808128,"data_stored":261808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:09:59.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:00.223 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:00.224 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:00.250 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-15T21:10:00.064909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513922862,"num_objects":62740,"num_object_clones":0,"num_object_copies":250960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137585,"ondisk_log_size":137585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743940,"kb_used_data":1004608,"kb_used_omap":14,"kb_used_meta":1739249,"kb_avail":374743420,"statfs":{"total":386547056640,"available":383737262080,"internally_reserved":0,"allocated":1028718592,"data_stored":1028058636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1780991538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,5,8,3,8,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21028864,"num_objects":-2567,"num_object_clones":0,"num_object_copies":-10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005031"},"pg_stats":[{"pgid":"1.f","version":"33'8467","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.412162+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:56.412162+0000","last_peered":"2024-09-15T21:09:56.412162+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:56.412162+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31825920,"num_objects":3885,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8671","reported_seq":12163,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.860886+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:55.860886+0000","last_peered":"2024-09-15T21:09:55.860886+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:55.860886+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31790348,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8519","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.252277+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:58.252277+0000","last_peered":"2024-09-15T21:09:58.252277+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:58.252277+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32053918,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8518,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8572","reported_seq":8642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.772583+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:55.772583+0000","last_peered":"2024-09-15T21:09:55.772583+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:55.772583+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31814924,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8657","reported_seq":8714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.467451+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:56.467451+0000","last_peered":"2024-09-15T21:09:56.467451+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:56.467451+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32781580,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8516","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.331281+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:56.331281+0000","last_peered":"2024-09-15T21:09:56.331281+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:56.331281+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31621120,"num_objects":3860,"num_object_clones":0,"num_object_copies":15440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8764","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908625+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:55.908625+0000","last_peered":"2024-09-15T21:09:55.908625+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:55.908625+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830732,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8866","reported_seq":8936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.689373+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:55.689373+0000","last_peered":"2024-09-15T21:09:55.689373+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:55.689373+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31905036,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8581","reported_seq":8642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.551095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:56.551095+0000","last_peered":"2024-09-15T21:09:56.551095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:56.551095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31705624,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8664","reported_seq":8746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.602241+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:58.602241+0000","last_peered":"2024-09-15T21:09:58.602241+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:58.602241+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929612,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8461","reported_seq":13125,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:56.448495+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:56.448495+0000","last_peered":"2024-09-15T21:09:56.448495+0000","last_clean":"2024-09-15T21:09:56.448495+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:56.448495+0000","last_undegraded":"2024-09-15T21:09:56.448495+0000","last_fullsized":"2024-09-15T21:09:56.448495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8461,"log_dups_size":0,"ondisk_log_size":8461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2572,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8484","reported_seq":8525,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.199271+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:56.199271+0000","last_peered":"2024-09-15T21:09:56.199271+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:56.199271+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32299654,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1687}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8585","reported_seq":8666,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.547611+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:58.547611+0000","last_peered":"2024-09-15T21:09:58.547611+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:58.547611+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078470,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8513","reported_seq":8571,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.468843+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:56.468843+0000","last_peered":"2024-09-15T21:09:56.468843+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:56.468843+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32101644,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1640}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8663","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.427259+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:58.427259+0000","last_peered":"2024-09-15T21:09:58.427259+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:58.427259+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912652,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8602","reported_seq":8672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908286+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:55.908286+0000","last_peered":"2024-09-15T21:09:55.908286+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:55.908286+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32331020,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513922862,"num_objects":62740,"num_object_clones":0,"num_object_copies":250960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025687552,"data_stored":1025687552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137585,"ondisk_log_size":137585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720796,"kb_used_data":251788,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93651044,"statfs":{"total":96636764160,"available":95898669056,"internally_reserved":0,"allocated":257830912,"data_stored":257665923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723076,"kb_used_data":250416,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93648764,"statfs":{"total":96636764160,"available":95896334336,"internally_reserved":0,"allocated":256425984,"data_stored":256260995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,0,1,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574388,"kb_used_data":249828,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93797452,"statfs":{"total":96636764160,"available":96048590848,"internally_reserved":0,"allocated":255823872,"data_stored":255658883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725680,"kb_used_data":252576,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93646160,"statfs":{"total":96636764160,"available":95893667840,"internally_reserved":0,"allocated":258637824,"data_stored":258472835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,5,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257732608,"data_stored":257732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255545344,"data_stored":255545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255221760,"data_stored":255221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257187840,"data_stored":257187840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:00.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:00.901 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:00.901 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:00.921 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-15T21:10:00.064909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":513922862,"num_objects":62740,"num_object_clones":0,"num_object_copies":250960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137585,"ondisk_log_size":137585,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2743940,"kb_used_data":1004608,"kb_used_omap":14,"kb_used_meta":1739249,"kb_avail":374743420,"statfs":{"total":386547056640,"available":383737262080,"internally_reserved":0,"allocated":1028718592,"data_stored":1028058636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1780991538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,8,5,8,3,8,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21028864,"num_objects":-2567,"num_object_clones":0,"num_object_copies":-10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005031"},"pg_stats":[{"pgid":"1.f","version":"33'8467","reported_seq":8513,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.412162+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:09:56.412162+0000","last_peered":"2024-09-15T21:09:56.412162+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:09:56.412162+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8467,"log_dups_size":0,"ondisk_log_size":8467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31825920,"num_objects":3885,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8467,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1609}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8671","reported_seq":12163,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.860886+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:09:55.860886+0000","last_peered":"2024-09-15T21:09:55.860886+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:09:55.860886+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8671,"log_dups_size":0,"ondisk_log_size":8671,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31790348,"num_objects":3881,"num_object_clones":0,"num_object_copies":15524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8669,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1715,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3212}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8519","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.252277+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:58.252277+0000","last_peered":"2024-09-15T21:09:58.252277+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:58.252277+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32053918,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8518,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8572","reported_seq":8642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.772583+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:09:55.772583+0000","last_peered":"2024-09-15T21:09:55.772583+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:09:55.772583+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8572,"log_dups_size":0,"ondisk_log_size":8572,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31814924,"num_objects":3884,"num_object_clones":0,"num_object_copies":15536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8570,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8657","reported_seq":8714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.467451+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:09:56.467451+0000","last_peered":"2024-09-15T21:09:56.467451+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:09:56.467451+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32781580,"num_objects":4002,"num_object_clones":0,"num_object_copies":16008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1691}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8516","reported_seq":8561,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.331281+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:09:56.331281+0000","last_peered":"2024-09-15T21:09:56.331281+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:09:56.331281+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8516,"log_dups_size":0,"ondisk_log_size":8516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31621120,"num_objects":3860,"num_object_clones":0,"num_object_copies":15440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8516,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1544}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8764","reported_seq":8835,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908625+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:09:55.908625+0000","last_peered":"2024-09-15T21:09:55.908625+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:09:55.908625+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8764,"log_dups_size":0,"ondisk_log_size":8764,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32830732,"num_objects":4008,"num_object_clones":0,"num_object_copies":16032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8762,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1642}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8866","reported_seq":8936,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.689373+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:09:55.689373+0000","last_peered":"2024-09-15T21:09:55.689373+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:09:55.689373+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31905036,"num_objects":3895,"num_object_clones":0,"num_object_copies":15580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1421}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8581","reported_seq":8642,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.551095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:09:56.551095+0000","last_peered":"2024-09-15T21:09:56.551095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:09:56.551095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31705624,"num_objects":3871,"num_object_clones":0,"num_object_copies":15484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8578,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1539}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8664","reported_seq":8746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.602241+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:58.602241+0000","last_peered":"2024-09-15T21:09:58.602241+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:58.602241+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929612,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8461","reported_seq":13125,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:09:56.448495+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:09:56.448495+0000","last_peered":"2024-09-15T21:09:56.448495+0000","last_clean":"2024-09-15T21:09:56.448495+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:09:56.448495+0000","last_undegraded":"2024-09-15T21:09:56.448495+0000","last_fullsized":"2024-09-15T21:09:56.448495+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8461,"log_dups_size":0,"ondisk_log_size":8461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31940608,"num_objects":3899,"num_object_clones":0,"num_object_copies":15596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2572,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8484","reported_seq":8525,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.199271+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:09:56.199271+0000","last_peered":"2024-09-15T21:09:56.199271+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:09:56.199271+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8484,"log_dups_size":0,"ondisk_log_size":8484,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32299654,"num_objects":3943,"num_object_clones":0,"num_object_copies":15772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8483,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1687}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8585","reported_seq":8666,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.547611+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:58.547611+0000","last_peered":"2024-09-15T21:09:58.547611+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:58.547611+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078470,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8513","reported_seq":8571,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:56.468843+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:09:56.468843+0000","last_peered":"2024-09-15T21:09:56.468843+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:09:56.468843+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8513,"log_dups_size":0,"ondisk_log_size":8513,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32101644,"num_objects":3919,"num_object_clones":0,"num_object_copies":15676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3919,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8511,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1640}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8663","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.427259+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:58.427259+0000","last_peered":"2024-09-15T21:09:58.427259+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:58.427259+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912652,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8602","reported_seq":8672,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:55.908286+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:09:55.908286+0000","last_peered":"2024-09-15T21:09:55.908286+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:09:55.908286+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8602,"log_dups_size":0,"ondisk_log_size":8602,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32331020,"num_objects":3947,"num_object_clones":0,"num_object_copies":15788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8600,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1632}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":513922862,"num_objects":62740,"num_object_clones":0,"num_object_copies":250960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62740,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131672,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1737,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1025687552,"data_stored":1025687552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137585,"ondisk_log_size":137585,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720796,"kb_used_data":251788,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93651044,"statfs":{"total":96636764160,"available":95898669056,"internally_reserved":0,"allocated":257830912,"data_stored":257665923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723076,"kb_used_data":250416,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93648764,"statfs":{"total":96636764160,"available":95896334336,"internally_reserved":0,"allocated":256425984,"data_stored":256260995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,0,1,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574388,"kb_used_data":249828,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93797452,"statfs":{"total":96636764160,"available":96048590848,"internally_reserved":0,"allocated":255823872,"data_stored":255658883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247894,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725680,"kb_used_data":252576,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93646160,"statfs":{"total":96636764160,"available":95893667840,"internally_reserved":0,"allocated":258637824,"data_stored":258472835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,5,1,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257732608,"data_stored":257732608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255545344,"data_stored":255545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255221760,"data_stored":255221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257187840,"data_stored":257187840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:00.922 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:00.922 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8467", 'reported_seq': 8513, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.412162+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:09:56.412162+0000', 'last_peered': '2024-09-15T21:09:56.412162+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:09:56.412162+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8467, 'log_dups_size': 0, 'ondisk_log_size': 8467, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31825920, 'num_objects': 3885, 'num_object_clones': 0, 'num_object_copies': 15540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8467, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1609}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8671", 'reported_seq': 12163, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:55.860886+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:09:55.860886+0000', 'last_peered': '2024-09-15T21:09:55.860886+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:09:55.860886+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8671, 'log_dups_size': 0, 'ondisk_log_size': 8671, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31790348, 'num_objects': 3881, 'num_object_clones': 0, 'num_object_copies': 15524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 669, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3881, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8669, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1715, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3212}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8519", 'reported_seq': 8600, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:58.252277+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:09:58.252277+0000', 'last_peered': '2024-09-15T21:09:58.252277+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:09:58.252277+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8519, 'log_dups_size': 0, 'ondisk_log_size': 8519, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32053918, 'num_objects': 3914, 'num_object_clones': 0, 'num_object_copies': 15656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3914, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8518, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1656}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8572", 'reported_seq': 8642, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:55.772583+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:09:55.772583+0000', 'last_peered': '2024-09-15T21:09:55.772583+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:09:55.772583+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8572, 'log_dups_size': 0, 'ondisk_log_size': 8572, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31814924, 'num_objects': 3884, 'num_object_clones': 0, 'num_object_copies': 15536, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3884, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8570, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8657", 'reported_seq': 8714, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.467451+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:09:56.467451+0000', 'last_peered': '2024-09-15T21:09:56.467451+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:09:56.467451+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8657, 'log_dups_size': 0, 'ondisk_log_size': 8657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32781580, 'num_objects': 4002, 'num_object_clones': 0, 'num_object_copies': 16008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1691}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:00.923 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8516", 'reported_seq': 8561, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.331281+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:09:56.331281+0000', 'last_peered': '2024-09-15T21:09:56.331281+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:09:56.331281+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8516, 'log_dups_size': 0, 'ondisk_log_size': 8516, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31621120, 'num_objects': 3860, 'num_object_clones': 0, 'num_object_copies': 15440, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3860, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8516, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1544}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8764", 'reported_seq': 8835, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:55.908625+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:09:55.908625+0000', 'last_peered': '2024-09-15T21:09:55.908625+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:09:55.908625+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8764, 'log_dups_size': 0, 'ondisk_log_size': 8764, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32830732, 'num_objects': 4008, 'num_object_clones': 0, 'num_object_copies': 16032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4008, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8762, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1642}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'8866", 'reported_seq': 8936, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:55.689373+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:09:55.689373+0000', 'last_peered': '2024-09-15T21:09:55.689373+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:09:55.689373+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8866, 'log_dups_size': 0, 'ondisk_log_size': 8866, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31905036, 'num_objects': 3895, 'num_object_clones': 0, 'num_object_copies': 15580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8864, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1421}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8581", 'reported_seq': 8642, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.551095+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:09:56.551095+0000', 'last_peered': '2024-09-15T21:09:56.551095+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:09:56.551095+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8581, 'log_dups_size': 0, 'ondisk_log_size': 8581, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31705624, 'num_objects': 3871, 'num_object_clones': 0, 'num_object_copies': 15484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3871, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8578, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1539}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8664", 'reported_seq': 8746, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:58.602241+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:09:58.602241+0000', 'last_peered': '2024-09-15T21:09:58.602241+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:09:58.602241+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8664, 'log_dups_size': 0, 'ondisk_log_size': 8664, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31929612, 'num_objects': 3898, 'num_object_clones': 0, 'num_object_copies': 15592, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3898, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8662, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1563}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8484", 'reported_seq': 8525, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.199271+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:09:56.199271+0000', 'last_peered': '2024-09-15T21:09:56.199271+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:09:56.199271+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8484, 'log_dups_size': 0, 'ondisk_log_size': 8484, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32299654, 'num_objects': 3943, 'num_object_clones': 0, 'num_object_copies': 15772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3943, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8483, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.924 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8585", 'reported_seq': 8666, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:58.547611+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:09:58.547611+0000', 'last_peered': '2024-09-15T21:09:58.547611+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:09:58.547611+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8585, 'log_dups_size': 0, 'ondisk_log_size': 8585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32078470, 'num_objects': 3916, 'num_object_clones': 0, 'num_object_copies': 15664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3916, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8584, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1630}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8513", 'reported_seq': 8571, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:56.468843+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:09:56.468843+0000', 'last_peered': '2024-09-15T21:09:56.468843+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:09:56.468843+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8513, 'log_dups_size': 0, 'ondisk_log_size': 8513, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32101644, 'num_objects': 3919, 'num_object_clones': 0, 'num_object_copies': 15676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3919, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8511, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1640}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8663", 'reported_seq': 8747, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:58.427259+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:09:58.427259+0000', 'last_peered': '2024-09-15T21:09:58.427259+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:09:58.427259+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8663, 'log_dups_size': 0, 'ondisk_log_size': 8663, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32912652, 'num_objects': 4018, 'num_object_clones': 0, 'num_object_copies': 16072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8661, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1737}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:00.925 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8602", 'reported_seq': 8672, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:09:55.908286+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:09:55.908286+0000', 'last_peered': '2024-09-15T21:09:55.908286+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:09:55.908286+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8602, 'log_dups_size': 0, 'ondisk_log_size': 8602, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32331020, 'num_objects': 3947, 'num_object_clones': 0, 'num_object_copies': 15788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3947, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8600, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1632}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:00.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:01.680 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:01.680 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 33110/250960 objects degraded (13.193%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":467,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":62740,"data_bytes":513922862,"bytes_used":2809794560,"bytes_avail":383737262080,"bytes_total":386547056640,"degraded_objects":33110,"degraded_total":250960,"degraded_ratio":0.13193337583678674,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":63461,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":761856,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":213},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:01.707 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:01.707 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:02.314 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:02.314 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:02.333 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-15T21:10:02.065760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":507000622,"num_objects":61895,"num_object_clones":0,"num_object_copies":247580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61895,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132517,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1810,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138430,"ondisk_log_size":138430,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726692,"kb_used_data":995672,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374760668,"statfs":{"total":386547056640,"available":383754924032,"internally_reserved":0,"allocated":1019568128,"data_stored":1018908172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1772471858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,4,6,6,6,11,17,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25427968,"num_objects":-3104,"num_object_clones":0,"num_object_copies":-12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":761856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005170"},"pg_stats":[{"pgid":"1.f","version":"33'8534","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.524580+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:01.524580+0000","last_peered":"2024-09-15T21:10:01.524580+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:01.524580+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8733","reported_seq":12374,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.952221+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:00.952221+0000","last_peered":"2024-09-15T21:10:00.952221+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:00.952221+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282444,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1788,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8519","reported_seq":8600,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.252277+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:09:58.252277+0000","last_peered":"2024-09-15T21:09:58.252277+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:09:58.252277+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8519,"log_dups_size":0,"ondisk_log_size":8519,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32053918,"num_objects":3914,"num_object_clones":0,"num_object_copies":15656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8518,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1656}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8644","reported_seq":8714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.719398+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:00.719398+0000","last_peered":"2024-09-15T21:10:00.719398+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:00.719398+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31225100,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8733","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.589005+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:01.589005+0000","last_peered":"2024-09-15T21:10:01.589005+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:01.589005+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158988,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8581","reported_seq":8624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.495348+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:01.495348+0000","last_peered":"2024-09-15T21:10:01.495348+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:01.495348+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088640,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8830","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.873313+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:00.873313+0000","last_peered":"2024-09-15T21:10:00.873313+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:00.873313+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32290060,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8937","reported_seq":9007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.755469+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:00.755469+0000","last_peered":"2024-09-15T21:10:00.755469+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:00.755469+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31323404,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8935,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8657","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.465661+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:01.465661+0000","last_peered":"2024-09-15T21:10:01.465661+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:01.465661+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31083032,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8664","reported_seq":8746,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.602241+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:09:58.602241+0000","last_peered":"2024-09-15T21:09:58.602241+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:09:58.602241+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8664,"log_dups_size":0,"ondisk_log_size":8664,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31929612,"num_objects":3898,"num_object_clones":0,"num_object_copies":15592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8662,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1563}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8544","reported_seq":13207,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:01.517595+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:01.517595+0000","last_peered":"2024-09-15T21:10:01.517595+0000","last_clean":"2024-09-15T21:10:01.517595+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:01.517595+0000","last_undegraded":"2024-09-15T21:10:01.517595+0000","last_fullsized":"2024-09-15T21:10:01.517595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.614138+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:01.614138+0000","last_peered":"2024-09-15T21:10:01.614138+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:01.614138+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783558,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8585","reported_seq":8666,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.547611+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:09:58.547611+0000","last_peered":"2024-09-15T21:09:58.547611+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:09:58.547611+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32078470,"num_objects":3916,"num_object_clones":0,"num_object_copies":15664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8584,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1630}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8579","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.471604+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:01.471604+0000","last_peered":"2024-09-15T21:10:01.471604+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:01.471604+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560972,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8663","reported_seq":8747,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:09:58.427259+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:09:58.427259+0000","last_peered":"2024-09-15T21:09:58.427259+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:09:58.427259+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32912652,"num_objects":4018,"num_object_clones":0,"num_object_copies":16072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8661,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1737}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8680","reported_seq":8748,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.951469+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:00.951469+0000","last_peered":"2024-09-15T21:10:00.951469+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:00.951469+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692044,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":507000622,"num_objects":61895,"num_object_clones":0,"num_object_copies":247580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61895,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132517,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1810,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1016524800,"data_stored":1016524800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138430,"ondisk_log_size":138430,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720484,"kb_used_data":247312,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93651356,"statfs":{"total":96636764160,"available":95898988544,"internally_reserved":0,"allocated":253247488,"data_stored":253082499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,4,6,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723076,"kb_used_data":250416,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":93648764,"statfs":{"total":96636764160,"available":95896334336,"internally_reserved":0,"allocated":256425984,"data_stored":256260995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,0,0,1,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574388,"kb_used_data":249828,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93797452,"statfs":{"total":96636764160,"available":96048590848,"internally_reserved":0,"allocated":255823872,"data_stored":255658883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708744,"kb_used_data":248116,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93663096,"statfs":{"total":96636764160,"available":95911010304,"internally_reserved":0,"allocated":254070784,"data_stored":253905795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,0,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253169664,"data_stored":253169664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255545344,"data_stored":255545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":255221760,"data_stored":255221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252588032,"data_stored":252588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:05.335 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:05.936 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:05.937 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:05.964 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-15T21:10:04.066610+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":504567598,"num_objects":61598,"num_object_clones":0,"num_object_copies":246392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132814,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1810,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138727,"ondisk_log_size":138727,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708984,"kb_used_data":990448,"kb_used_omap":14,"kb_used_meta":1718449,"kb_avail":374778376,"statfs":{"total":386547056640,"available":383773057024,"internally_reserved":0,"allocated":1014218752,"data_stored":1013558796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1759692338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,6,12,7,21,18,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21045248,"num_objects":-2569,"num_object_clones":0,"num_object_copies":-10276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2569,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005137"},"pg_stats":[{"pgid":"1.f","version":"33'8534","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.524580+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:01.524580+0000","last_peered":"2024-09-15T21:10:01.524580+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:01.524580+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8733","reported_seq":12374,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.952221+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:00.952221+0000","last_peered":"2024-09-15T21:10:00.952221+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:00.952221+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31282444,"num_objects":3819,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1788,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3223}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8589","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.621934+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:03.621934+0000","last_peered":"2024-09-15T21:10:03.621934+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:03.621934+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480478,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8588,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8644","reported_seq":8714,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.719398+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:00.719398+0000","last_peered":"2024-09-15T21:10:00.719398+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:00.719398+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8644,"log_dups_size":0,"ondisk_log_size":8644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31225100,"num_objects":3812,"num_object_clones":0,"num_object_copies":15248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8733","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.589005+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:01.589005+0000","last_peered":"2024-09-15T21:10:01.589005+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:01.589005+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158988,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8581","reported_seq":8624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.495348+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:01.495348+0000","last_peered":"2024-09-15T21:10:01.495348+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:01.495348+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088640,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8830","reported_seq":8901,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.873313+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:00.873313+0000","last_peered":"2024-09-15T21:10:00.873313+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:00.873313+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8830,"log_dups_size":0,"ondisk_log_size":8830,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32290060,"num_objects":3942,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8828,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1576}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'8937","reported_seq":9007,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.755469+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:00.755469+0000","last_peered":"2024-09-15T21:10:00.755469+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:00.755469+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8937,"log_dups_size":0,"ondisk_log_size":8937,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31323404,"num_objects":3824,"num_object_clones":0,"num_object_copies":15296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8935,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1350}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8657","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.465661+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:01.465661+0000","last_peered":"2024-09-15T21:10:01.465661+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:01.465661+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31083032,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8747","reported_seq":8829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.581360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:03.581360+0000","last_peered":"2024-09-15T21:10:03.581360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:03.581360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8747,"log_dups_size":0,"ondisk_log_size":8747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249676,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8544","reported_seq":13207,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:01.517595+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:01.517595+0000","last_peered":"2024-09-15T21:10:01.517595+0000","last_clean":"2024-09-15T21:10:01.517595+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:01.517595+0000","last_undegraded":"2024-09-15T21:10:01.517595+0000","last_fullsized":"2024-09-15T21:10:01.517595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.614138+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:01.614138+0000","last_peered":"2024-09-15T21:10:01.614138+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:01.614138+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783558,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8649","reported_seq":8730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.309082+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:03.309082+0000","last_peered":"2024-09-15T21:10:03.309082+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:03.309082+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8579","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.471604+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:01.471604+0000","last_peered":"2024-09-15T21:10:01.471604+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:01.471604+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560972,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8743","reported_seq":8823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.663926+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:03.663926+0000","last_peered":"2024-09-15T21:10:03.663926+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:03.663926+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257292,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8680","reported_seq":8748,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:00.951469+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:00.951469+0000","last_peered":"2024-09-15T21:10:00.951469+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:00.951469+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8680,"log_dups_size":0,"ondisk_log_size":8680,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31692044,"num_objects":3869,"num_object_clones":0,"num_object_copies":15476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8678,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1554}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":504567598,"num_objects":61598,"num_object_clones":0,"num_object_copies":246392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61598,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132814,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1810,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1012031488,"data_stored":1012031488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138727,"ondisk_log_size":138727,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720484,"kb_used_data":247312,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93651356,"statfs":{"total":96636764160,"available":95898988544,"internally_reserved":0,"allocated":253247488,"data_stored":253082499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,4,6,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":245192,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":251076608,"data_stored":250911619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,11,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051657,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574388,"kb_used_data":249828,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93797452,"statfs":{"total":96636764160,"available":96048590848,"internally_reserved":0,"allocated":255823872,"data_stored":255658883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247895,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708744,"kb_used_data":248116,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93663096,"statfs":{"total":96636764160,"available":95911010304,"internally_reserved":0,"allocated":254070784,"data_stored":253905795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":471658190},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,4,0,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":253169664,"data_stored":253169664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":255545344,"data_stored":255545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250728448,"data_stored":250728448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252588032,"data_stored":252588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:05.966 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:06.538 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:06.539 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:06.562 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-15T21:10:06.067350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":501749550,"num_objects":61254,"num_object_clones":0,"num_object_copies":245016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133158,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139071,"ondisk_log_size":139071,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705196,"kb_used_data":982496,"kb_used_omap":14,"kb_used_meta":1722609,"kb_avail":374782164,"statfs":{"total":386547056640,"available":383776935936,"internally_reserved":0,"allocated":1006075904,"data_stored":1005415948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1763952178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,6,11,22,17,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21471232,"num_objects":-2621,"num_object_clones":0,"num_object_copies":-10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005033"},"pg_stats":[{"pgid":"1.f","version":"33'8534","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.524580+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:01.524580+0000","last_peered":"2024-09-15T21:10:01.524580+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:01.524580+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8805","reported_seq":12583,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.993055+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:05.993055+0000","last_peered":"2024-09-15T21:10:05.993055+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:05.993055+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8589","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.621934+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:03.621934+0000","last_peered":"2024-09-15T21:10:03.621934+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:03.621934+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480478,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8588,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8709","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.934767+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:05.934767+0000","last_peered":"2024-09-15T21:10:05.934767+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:05.934767+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8733","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.589005+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:01.589005+0000","last_peered":"2024-09-15T21:10:01.589005+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:01.589005+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158988,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8581","reported_seq":8624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.495348+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:01.495348+0000","last_peered":"2024-09-15T21:10:01.495348+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:01.495348+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088640,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8893","reported_seq":8963,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.847641+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:05.847641+0000","last_peered":"2024-09-15T21:10:05.847641+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:05.847641+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31773964,"num_objects":3879,"num_object_clones":0,"num_object_copies":15516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9005","reported_seq":9069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.959764+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:05.959764+0000","last_peered":"2024-09-15T21:10:05.959764+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:05.959764+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30766348,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8657","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.465661+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:01.465661+0000","last_peered":"2024-09-15T21:10:01.465661+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:01.465661+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31083032,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8747","reported_seq":8829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.581360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:03.581360+0000","last_peered":"2024-09-15T21:10:03.581360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:03.581360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8747,"log_dups_size":0,"ondisk_log_size":8747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249676,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8544","reported_seq":13207,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:01.517595+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:01.517595+0000","last_peered":"2024-09-15T21:10:01.517595+0000","last_clean":"2024-09-15T21:10:01.517595+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:01.517595+0000","last_undegraded":"2024-09-15T21:10:01.517595+0000","last_fullsized":"2024-09-15T21:10:01.517595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.614138+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:01.614138+0000","last_peered":"2024-09-15T21:10:01.614138+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:01.614138+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783558,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8649","reported_seq":8730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.309082+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:03.309082+0000","last_peered":"2024-09-15T21:10:03.309082+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:03.309082+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8579","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.471604+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:01.471604+0000","last_peered":"2024-09-15T21:10:01.471604+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:01.471604+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560972,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8743","reported_seq":8823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.663926+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:03.663926+0000","last_peered":"2024-09-15T21:10:03.663926+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:03.663926+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257292,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8756","reported_seq":8826,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.771854+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:05.771854+0000","last_peered":"2024-09-15T21:10:05.771854+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:05.771854+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069452,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":501749550,"num_objects":61254,"num_object_clones":0,"num_object_copies":245016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133158,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1002950656,"data_stored":1002950656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139071,"ondisk_log_size":139071,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720484,"kb_used_data":247312,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93651356,"statfs":{"total":96636764160,"available":95898988544,"internally_reserved":0,"allocated":253247488,"data_stored":253082499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,4,6,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":245192,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":251076608,"data_stored":250911619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,11,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570952,"kb_used_data":246392,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93800888,"statfs":{"total":96636764160,"available":96052109312,"internally_reserved":0,"allocated":252305408,"data_stored":252140419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708392,"kb_used_data":243600,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93663448,"statfs":{"total":96636764160,"available":95911370752,"internally_reserved":0,"allocated":249446400,"data_stored":249281411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248578048,"data_stored":248578048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251056128,"data_stored":251056128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250728448,"data_stored":250728448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252588032,"data_stored":252588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8534", 'reported_seq': 8581, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.524580+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:01.524580+0000', 'last_peered': '2024-09-15T21:10:01.524580+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:01.524580+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8534, 'log_dups_size': 0, 'ondisk_log_size': 8534, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31277056, 'num_objects': 3818, 'num_object_clones': 0, 'num_object_copies': 15272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8534, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1542}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8805", 'reported_seq': 12583, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:05.993055+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:05.993055+0000', 'last_peered': '2024-09-15T21:10:05.993055+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:05.993055+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8805, 'log_dups_size': 0, 'ondisk_log_size': 8805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30692620, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 528, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8803, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1856, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8589", 'reported_seq': 8667, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:03.621934+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:03.621934+0000', 'last_peered': '2024-09-15T21:10:03.621934+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:03.621934+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8589, 'log_dups_size': 0, 'ondisk_log_size': 8589, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31480478, 'num_objects': 3844, 'num_object_clones': 0, 'num_object_copies': 15376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3844, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8588, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1586}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:06.563 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8709", 'reported_seq': 8779, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:05.934767+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:05.934767+0000', 'last_peered': '2024-09-15T21:10:05.934767+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:05.934767+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8709, 'log_dups_size': 0, 'ondisk_log_size': 8709, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30692620, 'num_objects': 3747, 'num_object_clones': 0, 'num_object_copies': 14988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8707, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1413}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8733", 'reported_seq': 8790, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.589005+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:01.589005+0000', 'last_peered': '2024-09-15T21:10:01.589005+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:01.589005+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8733, 'log_dups_size': 0, 'ondisk_log_size': 8733, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32158988, 'num_objects': 3926, 'num_object_clones': 0, 'num_object_copies': 15704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8731, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1615}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8581", 'reported_seq': 8624, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.495348+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:01.495348+0000', 'last_peered': '2024-09-15T21:10:01.495348+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:01.495348+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8581, 'log_dups_size': 0, 'ondisk_log_size': 8581, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31088640, 'num_objects': 3795, 'num_object_clones': 0, 'num_object_copies': 15180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8581, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1479}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8893", 'reported_seq': 8963, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:05.847641+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:05.847641+0000', 'last_peered': '2024-09-15T21:10:05.847641+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:05.847641+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8893, 'log_dups_size': 0, 'ondisk_log_size': 8893, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31773964, 'num_objects': 3879, 'num_object_clones': 0, 'num_object_copies': 15516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8891, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9005", 'reported_seq': 9069, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:05.959764+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:05.959764+0000', 'last_peered': '2024-09-15T21:10:05.959764+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:05.959764+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9005, 'log_dups_size': 0, 'ondisk_log_size': 9005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30766348, 'num_objects': 3756, 'num_object_clones': 0, 'num_object_copies': 15024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3756, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9003, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1282}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8657", 'reported_seq': 8718, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.465661+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:01.465661+0000', 'last_peered': '2024-09-15T21:10:01.465661+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:01.465661+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8657, 'log_dups_size': 0, 'ondisk_log_size': 8657, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31083032, 'num_objects': 3795, 'num_object_clones': 0, 'num_object_copies': 15180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3795, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8654, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1463}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.564 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8747", 'reported_seq': 8829, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:03.581360+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:03.581360+0000', 'last_peered': '2024-09-15T21:10:03.581360+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:03.581360+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8747, 'log_dups_size': 0, 'ondisk_log_size': 8747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31249676, 'num_objects': 3815, 'num_object_clones': 0, 'num_object_copies': 15260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3815, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8745, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1480}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8547", 'reported_seq': 8588, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.614138+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:01.614138+0000', 'last_peered': '2024-09-15T21:10:01.614138+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:01.614138+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8547, 'log_dups_size': 0, 'ondisk_log_size': 8547, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31783558, 'num_objects': 3880, 'num_object_clones': 0, 'num_object_copies': 15520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3880, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8546, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1624}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8649", 'reported_seq': 8730, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:03.309082+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:03.309082+0000', 'last_peered': '2024-09-15T21:10:03.309082+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:03.309082+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8649, 'log_dups_size': 0, 'ondisk_log_size': 8649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31554182, 'num_objects': 3852, 'num_object_clones': 0, 'num_object_copies': 15408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8648, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1566}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8579", 'reported_seq': 8637, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:01.471604+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:01.471604+0000', 'last_peered': '2024-09-15T21:10:01.471604+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:01.471604+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8579, 'log_dups_size': 0, 'ondisk_log_size': 8579, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31560972, 'num_objects': 3853, 'num_object_clones': 0, 'num_object_copies': 15412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3853, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8577, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8743", 'reported_seq': 8823, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:03.663926+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:03.663926+0000', 'last_peered': '2024-09-15T21:10:03.663926+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:03.663926+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8743, 'log_dups_size': 0, 'ondisk_log_size': 8743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 32257292, 'num_objects': 3938, 'num_object_clones': 0, 'num_object_copies': 15752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3938, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8741, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1657}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:06.565 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8756", 'reported_seq': 8826, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:05.771854+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:05.771854+0000', 'last_peered': '2024-09-15T21:10:05.771854+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:05.771854+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8756, 'log_dups_size': 0, 'ondisk_log_size': 8756, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31069452, 'num_objects': 3793, 'num_object_clones': 0, 'num_object_copies': 15172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3793, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8754, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:06.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:07.298 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:07.299 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32969/245016 objects degraded (13.456%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":473,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":61254,"data_bytes":501749550,"bytes_used":2770120704,"bytes_avail":383776935936,"bytes_total":386547056640,"degraded_objects":32969,"degraded_total":245016,"degraded_ratio":0.13455855944101611,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":27295,"recovering_keys_per_sec":0,"num_objects_recovered":189,"num_bytes_recovered":327680,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":218},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:07.340 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:07.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:07.943 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:07.944 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:07.968 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-15T21:10:06.067350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":501749550,"num_objects":61254,"num_object_clones":0,"num_object_copies":245016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133158,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139071,"ondisk_log_size":139071,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705196,"kb_used_data":982496,"kb_used_omap":14,"kb_used_meta":1722609,"kb_avail":374782164,"statfs":{"total":386547056640,"available":383776935936,"internally_reserved":0,"allocated":1006075904,"data_stored":1005415948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1763952178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,6,11,22,17,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21471232,"num_objects":-2621,"num_object_clones":0,"num_object_copies":-10484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2621,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005033"},"pg_stats":[{"pgid":"1.f","version":"33'8534","reported_seq":8581,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.524580+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:01.524580+0000","last_peered":"2024-09-15T21:10:01.524580+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:01.524580+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8534,"log_dups_size":0,"ondisk_log_size":8534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31277056,"num_objects":3818,"num_object_clones":0,"num_object_copies":15272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8805","reported_seq":12583,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.993055+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:05.993055+0000","last_peered":"2024-09-15T21:10:05.993055+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:05.993055+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8589","reported_seq":8667,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.621934+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:03.621934+0000","last_peered":"2024-09-15T21:10:03.621934+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:03.621934+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8589,"log_dups_size":0,"ondisk_log_size":8589,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31480478,"num_objects":3844,"num_object_clones":0,"num_object_copies":15376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8588,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1586}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8709","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.934767+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:05.934767+0000","last_peered":"2024-09-15T21:10:05.934767+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:05.934767+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8733","reported_seq":8790,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.589005+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:01.589005+0000","last_peered":"2024-09-15T21:10:01.589005+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:01.589005+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8733,"log_dups_size":0,"ondisk_log_size":8733,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32158988,"num_objects":3926,"num_object_clones":0,"num_object_copies":15704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8731,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8581","reported_seq":8624,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.495348+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:01.495348+0000","last_peered":"2024-09-15T21:10:01.495348+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:01.495348+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8581,"log_dups_size":0,"ondisk_log_size":8581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31088640,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8581,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8893","reported_seq":8963,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.847641+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:05.847641+0000","last_peered":"2024-09-15T21:10:05.847641+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:05.847641+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31773964,"num_objects":3879,"num_object_clones":0,"num_object_copies":15516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9005","reported_seq":9069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.959764+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:05.959764+0000","last_peered":"2024-09-15T21:10:05.959764+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:05.959764+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30766348,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8657","reported_seq":8718,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.465661+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:01.465661+0000","last_peered":"2024-09-15T21:10:01.465661+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:01.465661+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8657,"log_dups_size":0,"ondisk_log_size":8657,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31083032,"num_objects":3795,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8654,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1463}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8747","reported_seq":8829,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.581360+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:03.581360+0000","last_peered":"2024-09-15T21:10:03.581360+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:03.581360+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8747,"log_dups_size":0,"ondisk_log_size":8747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31249676,"num_objects":3815,"num_object_clones":0,"num_object_copies":15260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8745,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1480}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8544","reported_seq":13207,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:01.517595+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:01.517595+0000","last_peered":"2024-09-15T21:10:01.517595+0000","last_clean":"2024-09-15T21:10:01.517595+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:01.517595+0000","last_undegraded":"2024-09-15T21:10:01.517595+0000","last_fullsized":"2024-09-15T21:10:01.517595+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8544,"log_dups_size":0,"ondisk_log_size":8544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":3816,"num_object_clones":0,"num_object_copies":15264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2655,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8547","reported_seq":8588,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.614138+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:01.614138+0000","last_peered":"2024-09-15T21:10:01.614138+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:01.614138+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8547,"log_dups_size":0,"ondisk_log_size":8547,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31783558,"num_objects":3880,"num_object_clones":0,"num_object_copies":15520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8546,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1624}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8649","reported_seq":8730,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.309082+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:03.309082+0000","last_peered":"2024-09-15T21:10:03.309082+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:03.309082+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8649,"log_dups_size":0,"ondisk_log_size":8649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31554182,"num_objects":3852,"num_object_clones":0,"num_object_copies":15408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8579","reported_seq":8637,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:01.471604+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:01.471604+0000","last_peered":"2024-09-15T21:10:01.471604+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:01.471604+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8579,"log_dups_size":0,"ondisk_log_size":8579,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31560972,"num_objects":3853,"num_object_clones":0,"num_object_copies":15412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8577,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8743","reported_seq":8823,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:03.663926+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:03.663926+0000","last_peered":"2024-09-15T21:10:03.663926+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:03.663926+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8743,"log_dups_size":0,"ondisk_log_size":8743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32257292,"num_objects":3938,"num_object_clones":0,"num_object_copies":15752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8741,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1657}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8756","reported_seq":8826,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.771854+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:05.771854+0000","last_peered":"2024-09-15T21:10:05.771854+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:05.771854+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069452,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":501749550,"num_objects":61254,"num_object_clones":0,"num_object_copies":245016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61254,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133158,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1002950656,"data_stored":1002950656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139071,"ondisk_log_size":139071,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720484,"kb_used_data":247312,"kb_used_omap":4,"kb_used_meta":473147,"kb_avail":93651356,"statfs":{"total":96636764160,"available":95898988544,"internally_reserved":0,"allocated":253247488,"data_stored":253082499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":484503248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,4,6,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705368,"kb_used_data":245192,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93666472,"statfs":{"total":96636764160,"available":95914467328,"internally_reserved":0,"allocated":251076608,"data_stored":250911619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,2,11,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051658,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570952,"kb_used_data":246392,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93800888,"statfs":{"total":96636764160,"available":96052109312,"internally_reserved":0,"allocated":252305408,"data_stored":252140419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708392,"kb_used_data":243600,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93663448,"statfs":{"total":96636764160,"available":95911370752,"internally_reserved":0,"allocated":249446400,"data_stored":249281411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248578048,"data_stored":248578048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":251056128,"data_stored":251056128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":250728448,"data_stored":250728448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":252588032,"data_stored":252588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:10.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:11.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:11.552 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:11.578 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-15T21:10:10.068733+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":495474478,"num_objects":60488,"num_object_clones":0,"num_object_copies":241952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133924,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139837,"ondisk_log_size":139837,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704424,"kb_used_data":969240,"kb_used_omap":14,"kb_used_meta":1735089,"kb_avail":374782936,"statfs":{"total":386547056640,"available":383777726464,"internally_reserved":0,"allocated":992501760,"data_stored":991841804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1776731698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,0,2,12,16,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20676608,"num_objects":-2524,"num_object_clones":0,"num_object_copies":-10096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004752"},"pg_stats":[{"pgid":"1.f","version":"33'8593","reported_seq":8640,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.606111+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:06.606111+0000","last_peered":"2024-09-15T21:10:06.606111+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:06.606111+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8593,"log_dups_size":0,"ondisk_log_size":8593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30793728,"num_objects":3759,"num_object_clones":0,"num_object_copies":15036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8593,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1483}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8805","reported_seq":12583,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.993055+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:05.993055+0000","last_peered":"2024-09-15T21:10:05.993055+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:05.993055+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8805,"log_dups_size":0,"ondisk_log_size":8805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8803,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3219}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8654","reported_seq":8735,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.718189+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:08.718189+0000","last_peered":"2024-09-15T21:10:08.718189+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:08.718189+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30947998,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8653,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8709","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.934767+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:05.934767+0000","last_peered":"2024-09-15T21:10:05.934767+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:05.934767+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8709,"log_dups_size":0,"ondisk_log_size":8709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30692620,"num_objects":3747,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8707,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1413}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8795","reported_seq":8852,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.526513+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:06.526513+0000","last_peered":"2024-09-15T21:10:06.526513+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:06.526513+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8795,"log_dups_size":0,"ondisk_log_size":8795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31651084,"num_objects":3864,"num_object_clones":0,"num_object_copies":15456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8793,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8653","reported_seq":8699,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.345300+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:06.345300+0000","last_peered":"2024-09-15T21:10:06.345300+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:06.345300+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8653,"log_dups_size":0,"ondisk_log_size":8653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30498816,"num_objects":3723,"num_object_clones":0,"num_object_copies":14892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1407}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8893","reported_seq":8963,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.847641+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:05.847641+0000","last_peered":"2024-09-15T21:10:05.847641+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:05.847641+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8893,"log_dups_size":0,"ondisk_log_size":8893,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31773964,"num_objects":3879,"num_object_clones":0,"num_object_copies":15516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1513}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9005","reported_seq":9069,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.959764+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:05.959764+0000","last_peered":"2024-09-15T21:10:05.959764+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:05.959764+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9005,"log_dups_size":0,"ondisk_log_size":9005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30766348,"num_objects":3756,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1282}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8720","reported_seq":8779,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.732514+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:06.732514+0000","last_peered":"2024-09-15T21:10:06.732514+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:06.732514+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8720,"log_dups_size":0,"ondisk_log_size":8720,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30566936,"num_objects":3732,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8717,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1400}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8819","reported_seq":8900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.669081+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:08.669081+0000","last_peered":"2024-09-15T21:10:08.669081+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:08.669081+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30659852,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8618","reported_seq":13282,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:06.693355+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:06.693355+0000","last_peered":"2024-09-15T21:10:06.693355+0000","last_clean":"2024-09-15T21:10:06.693355+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:06.693355+0000","last_undegraded":"2024-09-15T21:10:06.693355+0000","last_fullsized":"2024-09-15T21:10:06.693355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8618,"log_dups_size":0,"ondisk_log_size":8618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30654464,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8621","reported_seq":8662,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.469303+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:06.469303+0000","last_peered":"2024-09-15T21:10:06.469303+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:06.469303+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8621,"log_dups_size":0,"ondisk_log_size":8621,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31177350,"num_objects":3806,"num_object_clones":0,"num_object_copies":15224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8620,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8728","reported_seq":8809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.450430+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:08.450430+0000","last_peered":"2024-09-15T21:10:08.450430+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:08.450430+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30907014,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8650","reported_seq":8708,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:06.530326+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:06.530326+0000","last_peered":"2024-09-15T21:10:06.530326+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:06.530326+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8650,"log_dups_size":0,"ondisk_log_size":8650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30979340,"num_objects":3782,"num_object_clones":0,"num_object_copies":15128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8648,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1503}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8818","reported_seq":8902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.699165+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:08.699165+0000","last_peered":"2024-09-15T21:10:08.699165+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:08.699165+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642892,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8756","reported_seq":8826,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:05.771854+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:05.771854+0000","last_peered":"2024-09-15T21:10:05.771854+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:05.771854+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8756,"log_dups_size":0,"ondisk_log_size":8756,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31069452,"num_objects":3793,"num_object_clones":0,"num_object_copies":15172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":495474478,"num_objects":60488,"num_object_clones":0,"num_object_copies":241952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32969,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60488,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133924,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1878,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":989417472,"data_stored":989417472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139837,"ondisk_log_size":139837,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720124,"kb_used_data":242796,"kb_used_omap":4,"kb_used_meta":477307,"kb_avail":93651716,"statfs":{"total":96636764160,"available":95899357184,"internally_reserved":0,"allocated":248623104,"data_stored":248458115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":488763088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705204,"kb_used_data":240868,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93666636,"statfs":{"total":96636764160,"available":95914635264,"internally_reserved":0,"allocated":246648832,"data_stored":246483843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570704,"kb_used_data":241976,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93801136,"statfs":{"total":96636764160,"available":96052363264,"internally_reserved":0,"allocated":247783424,"data_stored":247618435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247896,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708392,"kb_used_data":243600,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93663448,"statfs":{"total":96636764160,"available":95911370752,"internally_reserved":0,"allocated":249446400,"data_stored":249281411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":475918030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248578048,"data_stored":248578048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246640640,"data_stored":246640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246247424,"data_stored":246247424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":247951360,"data_stored":247951360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:11.579 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:12.216 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:12.216 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:12.246 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-15T21:10:12.069389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488748846,"num_objects":59667,"num_object_clones":0,"num_object_copies":238668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134745,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140658,"ondisk_log_size":140658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687384,"kb_used_data":960520,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374799976,"statfs":{"total":386547056640,"available":383795175424,"internally_reserved":0,"allocated":983572480,"data_stored":982912524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1768212018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,13,20,21,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25174016,"num_objects":-3073,"num_object_clones":0,"num_object_copies":-12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004479"},"pg_stats":[{"pgid":"1.f","version":"33'8668","reported_seq":8715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.601949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:11.601949+0000","last_peered":"2024-09-15T21:10:11.601949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:11.601949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30179328,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8874","reported_seq":12822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.093196+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:11.093196+0000","last_peered":"2024-09-15T21:10:11.093196+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:11.093196+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127372,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8654","reported_seq":8735,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.718189+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:08.718189+0000","last_peered":"2024-09-15T21:10:08.718189+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:08.718189+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30947998,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8653,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8783","reported_seq":8853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.067072+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:11.067072+0000","last_peered":"2024-09-15T21:10:11.067072+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:11.067072+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30086412,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8875","reported_seq":8931,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.732197+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:11.732197+0000","last_peered":"2024-09-15T21:10:11.732197+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:11.732197+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8721","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.642269+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:11.642269+0000","last_peered":"2024-09-15T21:10:11.642269+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:11.642269+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8955","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.809038+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:10.809038+0000","last_peered":"2024-09-15T21:10:10.809038+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:10.809038+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31266060,"num_objects":3817,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9056","reported_seq":9126,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.851975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:10.851975+0000","last_peered":"2024-09-15T21:10:10.851975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:10.851975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348556,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8780","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.645046+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:11.645046+0000","last_peered":"2024-09-15T21:10:11.645046+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:11.645046+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075416,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8819","reported_seq":8900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.669081+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:08.669081+0000","last_peered":"2024-09-15T21:10:08.669081+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:08.669081+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30659852,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8682","reported_seq":13346,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:11.767747+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:11.767747+0000","last_peered":"2024-09-15T21:10:11.767747+0000","last_clean":"2024-09-15T21:10:11.767747+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:11.767747+0000","last_undegraded":"2024-09-15T21:10:11.767747+0000","last_fullsized":"2024-09-15T21:10:11.767747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8685","reported_seq":8726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.704002+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:11.704002+0000","last_peered":"2024-09-15T21:10:11.704002+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:11.704002+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30653062,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8728","reported_seq":8809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.450430+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:08.450430+0000","last_peered":"2024-09-15T21:10:08.450430+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:08.450430+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30907014,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8724","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.669197+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:11.669197+0000","last_peered":"2024-09-15T21:10:11.669197+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:11.669197+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373132,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8818","reported_seq":8902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.699165+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:08.699165+0000","last_peered":"2024-09-15T21:10:08.699165+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:08.699165+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642892,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8836","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.841287+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:10.841287+0000","last_peered":"2024-09-15T21:10:10.841287+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:10.841287+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":488748846,"num_objects":59667,"num_object_clones":0,"num_object_copies":238668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134745,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":980422656,"data_stored":980422656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140658,"ondisk_log_size":140658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":238356,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":244076544,"data_stored":243911555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705204,"kb_used_data":240868,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93666636,"statfs":{"total":96636764160,"available":95914635264,"internally_reserved":0,"allocated":246648832,"data_stored":246483843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570704,"kb_used_data":241976,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93801136,"statfs":{"total":96636764160,"available":96052363264,"internally_reserved":0,"allocated":247783424,"data_stored":247618435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708272,"kb_used_data":239320,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93663568,"statfs":{"total":96636764160,"available":95911493632,"internally_reserved":0,"allocated":245063680,"data_stored":244898691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244035584,"data_stored":244035584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246640640,"data_stored":246640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246247424,"data_stored":246247424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":243499008,"data_stored":243499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8668", 'reported_seq': 8715, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.601949+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:11.601949+0000', 'last_peered': '2024-09-15T21:10:11.601949+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:11.601949+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8668, 'log_dups_size': 0, 'ondisk_log_size': 8668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30179328, 'num_objects': 3684, 'num_object_clones': 0, 'num_object_copies': 14736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8668, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8874", 'reported_seq': 12822, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.093196+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:11.093196+0000', 'last_peered': '2024-09-15T21:10:11.093196+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:11.093196+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8874, 'log_dups_size': 0, 'ondisk_log_size': 8874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30127372, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 443, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8872, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1941, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8654", 'reported_seq': 8735, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:08.718189+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:08.718189+0000', 'last_peered': '2024-09-15T21:10:08.718189+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:08.718189+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8654, 'log_dups_size': 0, 'ondisk_log_size': 8654, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30947998, 'num_objects': 3779, 'num_object_clones': 0, 'num_object_copies': 15116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3779, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8653, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1521}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:12.248 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8783", 'reported_seq': 8853, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.067072+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:11.067072+0000', 'last_peered': '2024-09-15T21:10:11.067072+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:11.067072+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8783, 'log_dups_size': 0, 'ondisk_log_size': 8783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30086412, 'num_objects': 3673, 'num_object_clones': 0, 'num_object_copies': 14692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8781, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8875", 'reported_seq': 8931, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.732197+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:11.732197+0000', 'last_peered': '2024-09-15T21:10:11.732197+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:11.732197+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8875, 'log_dups_size': 0, 'ondisk_log_size': 8875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30995724, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8873, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1473}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8721", 'reported_seq': 8767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.642269+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:11.642269+0000', 'last_peered': '2024-09-15T21:10:11.642269+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:11.642269+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8721, 'log_dups_size': 0, 'ondisk_log_size': 8721, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29941760, 'num_objects': 3655, 'num_object_clones': 0, 'num_object_copies': 14620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8721, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8955", 'reported_seq': 9026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.809038+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:10.809038+0000', 'last_peered': '2024-09-15T21:10:10.809038+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:10.809038+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8955, 'log_dups_size': 0, 'ondisk_log_size': 8955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31266060, 'num_objects': 3817, 'num_object_clones': 0, 'num_object_copies': 15268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8953, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1451}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9056", 'reported_seq': 9126, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.851975+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:10.851975+0000', 'last_peered': '2024-09-15T21:10:10.851975+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:10.851975+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9056, 'log_dups_size': 0, 'ondisk_log_size': 9056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30348556, 'num_objects': 3705, 'num_object_clones': 0, 'num_object_copies': 14820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9054, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8780", 'reported_seq': 8841, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.645046+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:11.645046+0000', 'last_peered': '2024-09-15T21:10:11.645046+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:11.645046+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8780, 'log_dups_size': 0, 'ondisk_log_size': 8780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30075416, 'num_objects': 3672, 'num_object_clones': 0, 'num_object_copies': 14688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8777, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.249 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8819", 'reported_seq': 8900, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:08.669081+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:08.669081+0000', 'last_peered': '2024-09-15T21:10:08.669081+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:08.669081+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8819, 'log_dups_size': 0, 'ondisk_log_size': 8819, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30659852, 'num_objects': 3743, 'num_object_clones': 0, 'num_object_copies': 14972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3743, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8817, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8685", 'reported_seq': 8726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.704002+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:11.704002+0000', 'last_peered': '2024-09-15T21:10:11.704002+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:11.704002+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8685, 'log_dups_size': 0, 'ondisk_log_size': 8685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30653062, 'num_objects': 3742, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8684, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8728", 'reported_seq': 8809, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:08.450430+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:08.450430+0000', 'last_peered': '2024-09-15T21:10:08.450430+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:08.450430+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8728, 'log_dups_size': 0, 'ondisk_log_size': 8728, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30907014, 'num_objects': 3773, 'num_object_clones': 0, 'num_object_copies': 15092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8727, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1487}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8724", 'reported_seq': 8782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.669197+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:11.669197+0000', 'last_peered': '2024-09-15T21:10:11.669197+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:11.669197+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8724, 'log_dups_size': 0, 'ondisk_log_size': 8724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30373132, 'num_objects': 3708, 'num_object_clones': 0, 'num_object_copies': 14832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8722, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8818", 'reported_seq': 8902, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:08.699165+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:08.699165+0000', 'last_peered': '2024-09-15T21:10:08.699165+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:08.699165+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8818, 'log_dups_size': 0, 'ondisk_log_size': 8818, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31642892, 'num_objects': 3863, 'num_object_clones': 0, 'num_object_copies': 15452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8816, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1582}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:12.250 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8836", 'reported_seq': 8906, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.841287+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:10.841287+0000', 'last_peered': '2024-09-15T21:10:10.841287+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:10.841287+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8836, 'log_dups_size': 0, 'ondisk_log_size': 8836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30414092, 'num_objects': 3713, 'num_object_clones': 0, 'num_object_copies': 14852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:12.251 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:13.044 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:13.044 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32884/238668 objects degraded (13.778%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":478,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":59667,"data_bytes":488748846,"bytes_used":2751881216,"bytes_avail":383795175424,"bytes_total":386547056640,"degraded_objects":32884,"degraded_total":238668,"degraded_ratio":0.1377813531767979,"recovering_objects_per_sec":18,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":255},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:13.070 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:13.070 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:13.741 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:13.742 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:13.762 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-15T21:10:12.069389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488748846,"num_objects":59667,"num_object_clones":0,"num_object_copies":238668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134745,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140658,"ondisk_log_size":140658,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2687384,"kb_used_data":960520,"kb_used_omap":14,"kb_used_meta":1726769,"kb_avail":374799976,"statfs":{"total":386547056640,"available":383795175424,"internally_reserved":0,"allocated":983572480,"data_stored":982912524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1768212018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,5,13,20,21,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-25174016,"num_objects":-3073,"num_object_clones":0,"num_object_copies":-12292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004479"},"pg_stats":[{"pgid":"1.f","version":"33'8668","reported_seq":8715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.601949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:11.601949+0000","last_peered":"2024-09-15T21:10:11.601949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:11.601949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30179328,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8874","reported_seq":12822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.093196+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:11.093196+0000","last_peered":"2024-09-15T21:10:11.093196+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:11.093196+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127372,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8654","reported_seq":8735,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.718189+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:08.718189+0000","last_peered":"2024-09-15T21:10:08.718189+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:08.718189+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8654,"log_dups_size":0,"ondisk_log_size":8654,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30947998,"num_objects":3779,"num_object_clones":0,"num_object_copies":15116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8653,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1521}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8783","reported_seq":8853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.067072+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:11.067072+0000","last_peered":"2024-09-15T21:10:11.067072+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:11.067072+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30086412,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8875","reported_seq":8931,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.732197+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:11.732197+0000","last_peered":"2024-09-15T21:10:11.732197+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:11.732197+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8721","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.642269+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:11.642269+0000","last_peered":"2024-09-15T21:10:11.642269+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:11.642269+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8955","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.809038+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:10.809038+0000","last_peered":"2024-09-15T21:10:10.809038+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:10.809038+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31266060,"num_objects":3817,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9056","reported_seq":9126,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.851975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:10.851975+0000","last_peered":"2024-09-15T21:10:10.851975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:10.851975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348556,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8780","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.645046+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:11.645046+0000","last_peered":"2024-09-15T21:10:11.645046+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:11.645046+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075416,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8819","reported_seq":8900,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.669081+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:08.669081+0000","last_peered":"2024-09-15T21:10:08.669081+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:08.669081+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8819,"log_dups_size":0,"ondisk_log_size":8819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30659852,"num_objects":3743,"num_object_clones":0,"num_object_copies":14972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1408}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8682","reported_seq":13346,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:11.767747+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:11.767747+0000","last_peered":"2024-09-15T21:10:11.767747+0000","last_clean":"2024-09-15T21:10:11.767747+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:11.767747+0000","last_undegraded":"2024-09-15T21:10:11.767747+0000","last_fullsized":"2024-09-15T21:10:11.767747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8685","reported_seq":8726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.704002+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:11.704002+0000","last_peered":"2024-09-15T21:10:11.704002+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:11.704002+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30653062,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8728","reported_seq":8809,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.450430+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:08.450430+0000","last_peered":"2024-09-15T21:10:08.450430+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:08.450430+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8728,"log_dups_size":0,"ondisk_log_size":8728,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30907014,"num_objects":3773,"num_object_clones":0,"num_object_copies":15092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8727,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1487}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8724","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.669197+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:11.669197+0000","last_peered":"2024-09-15T21:10:11.669197+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:11.669197+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373132,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8818","reported_seq":8902,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:08.699165+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:08.699165+0000","last_peered":"2024-09-15T21:10:08.699165+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:08.699165+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8818,"log_dups_size":0,"ondisk_log_size":8818,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31642892,"num_objects":3863,"num_object_clones":0,"num_object_copies":15452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8816,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1582}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8836","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.841287+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:10.841287+0000","last_peered":"2024-09-15T21:10:10.841287+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:10.841287+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":488748846,"num_objects":59667,"num_object_clones":0,"num_object_copies":238668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59667,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134745,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":980422656,"data_stored":980422656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140658,"ondisk_log_size":140658,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":238356,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":244076544,"data_stored":243911555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705204,"kb_used_data":240868,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93666636,"statfs":{"total":96636764160,"available":95914635264,"internally_reserved":0,"allocated":246648832,"data_stored":246483843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,4,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570704,"kb_used_data":241976,"kb_used_omap":4,"kb_used_meta":328699,"kb_avail":93801136,"statfs":{"total":96636764160,"available":96052363264,"internally_reserved":0,"allocated":247783424,"data_stored":247618435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":336588495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708272,"kb_used_data":239320,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93663568,"statfs":{"total":96636764160,"available":95911493632,"internally_reserved":0,"allocated":245063680,"data_stored":244898691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244035584,"data_stored":244035584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":246640640,"data_stored":246640640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":246247424,"data_stored":246247424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":243499008,"data_stored":243499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:16.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:17.347 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:17.347 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:17.371 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-15T21:10:16.071166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486709038,"num_objects":59418,"num_object_clones":0,"num_object_copies":237672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134994,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140907,"ondisk_log_size":140907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673812,"kb_used_data":951108,"kb_used_omap":14,"kb_used_meta":1722609,"kb_avail":374813548,"statfs":{"total":386547056640,"available":383809073152,"internally_reserved":0,"allocated":973934592,"data_stored":973274636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1763952178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,6,15,18,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17858560,"num_objects":-2180,"num_object_clones":0,"num_object_copies":-8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004555"},"pg_stats":[{"pgid":"1.f","version":"33'8668","reported_seq":8715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.601949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:11.601949+0000","last_peered":"2024-09-15T21:10:11.601949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:11.601949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30179328,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8874","reported_seq":12822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.093196+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:11.093196+0000","last_peered":"2024-09-15T21:10:11.093196+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:11.093196+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127372,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8699","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.706694+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:13.706694+0000","last_peered":"2024-09-15T21:10:13.706694+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:13.706694+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579358,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8698,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8783","reported_seq":8853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.067072+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:11.067072+0000","last_peered":"2024-09-15T21:10:11.067072+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:11.067072+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30086412,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8875","reported_seq":8931,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.732197+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:11.732197+0000","last_peered":"2024-09-15T21:10:11.732197+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:11.732197+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8721","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.642269+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:11.642269+0000","last_peered":"2024-09-15T21:10:11.642269+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:11.642269+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8955","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.809038+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:10.809038+0000","last_peered":"2024-09-15T21:10:10.809038+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:10.809038+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31266060,"num_objects":3817,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9056","reported_seq":9126,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.851975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:10.851975+0000","last_peered":"2024-09-15T21:10:10.851975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:10.851975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348556,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8780","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.645046+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:11.645046+0000","last_peered":"2024-09-15T21:10:11.645046+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:11.645046+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075416,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8890","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.705406+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:13.705406+0000","last_peered":"2024-09-15T21:10:13.705406+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:13.705406+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30078220,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8682","reported_seq":13346,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:11.767747+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:11.767747+0000","last_peered":"2024-09-15T21:10:11.767747+0000","last_clean":"2024-09-15T21:10:11.767747+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:11.767747+0000","last_undegraded":"2024-09-15T21:10:11.767747+0000","last_fullsized":"2024-09-15T21:10:11.767747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8685","reported_seq":8726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.704002+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:11.704002+0000","last_peered":"2024-09-15T21:10:11.704002+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:11.704002+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30653062,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8792","reported_seq":8873,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.556653+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:13.556653+0000","last_peered":"2024-09-15T21:10:13.556653+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:13.556653+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30382726,"num_objects":3709,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8724","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.669197+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:11.669197+0000","last_peered":"2024-09-15T21:10:11.669197+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:11.669197+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373132,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.408080+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:13.408080+0000","last_peered":"2024-09-15T21:10:13.408080+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:13.408080+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077644,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8836","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.841287+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:10.841287+0000","last_peered":"2024-09-15T21:10:10.841287+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:10.841287+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":486709038,"num_objects":59418,"num_object_clones":0,"num_object_copies":237672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134994,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971612160,"data_stored":971612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140907,"ondisk_log_size":140907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":238356,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":244076544,"data_stored":243911555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705100,"kb_used_data":236600,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93666740,"statfs":{"total":96636764160,"available":95914741760,"internally_reserved":0,"allocated":242278400,"data_stored":242113411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557236,"kb_used_data":236832,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93814604,"statfs":{"total":96636764160,"available":96066154496,"internally_reserved":0,"allocated":242515968,"data_stored":242350979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708272,"kb_used_data":239320,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93663568,"statfs":{"total":96636764160,"available":95911493632,"internally_reserved":0,"allocated":245063680,"data_stored":244898691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244035584,"data_stored":244035584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242208768,"data_stored":242208768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241868800,"data_stored":241868800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":243499008,"data_stored":243499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:17.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:17.987 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:17.987 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:18.006 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":265,"stamp":"2024-09-15T21:10:16.071166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486709038,"num_objects":59418,"num_object_clones":0,"num_object_copies":237672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134994,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140907,"ondisk_log_size":140907,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673812,"kb_used_data":951108,"kb_used_omap":14,"kb_used_meta":1722609,"kb_avail":374813548,"statfs":{"total":386547056640,"available":383809073152,"internally_reserved":0,"allocated":973934592,"data_stored":973274636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1763952178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,6,15,18,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17858560,"num_objects":-2180,"num_object_clones":0,"num_object_copies":-8720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004555"},"pg_stats":[{"pgid":"1.f","version":"33'8668","reported_seq":8715,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.601949+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:11.601949+0000","last_peered":"2024-09-15T21:10:11.601949+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:11.601949+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8668,"log_dups_size":0,"ondisk_log_size":8668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30179328,"num_objects":3684,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1408}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8874","reported_seq":12822,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.093196+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:11.093196+0000","last_peered":"2024-09-15T21:10:11.093196+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:11.093196+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30127372,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":443,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3235}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8699","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.706694+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:13.706694+0000","last_peered":"2024-09-15T21:10:13.706694+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:13.706694+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579358,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8698,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8783","reported_seq":8853,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.067072+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:11.067072+0000","last_peered":"2024-09-15T21:10:11.067072+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:11.067072+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8783,"log_dups_size":0,"ondisk_log_size":8783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30086412,"num_objects":3673,"num_object_clones":0,"num_object_copies":14692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8781,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8875","reported_seq":8931,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.732197+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:11.732197+0000","last_peered":"2024-09-15T21:10:11.732197+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:11.732197+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8875,"log_dups_size":0,"ondisk_log_size":8875,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30995724,"num_objects":3784,"num_object_clones":0,"num_object_copies":15136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8873,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1473}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8721","reported_seq":8767,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.642269+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:11.642269+0000","last_peered":"2024-09-15T21:10:11.642269+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:11.642269+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8721,"log_dups_size":0,"ondisk_log_size":8721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29941760,"num_objects":3655,"num_object_clones":0,"num_object_copies":14620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8721,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1339}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'8955","reported_seq":9026,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.809038+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:10.809038+0000","last_peered":"2024-09-15T21:10:10.809038+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:10.809038+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8955,"log_dups_size":0,"ondisk_log_size":8955,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31266060,"num_objects":3817,"num_object_clones":0,"num_object_copies":15268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8953,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1451}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9056","reported_seq":9126,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.851975+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:10.851975+0000","last_peered":"2024-09-15T21:10:10.851975+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:10.851975+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9056,"log_dups_size":0,"ondisk_log_size":9056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30348556,"num_objects":3705,"num_object_clones":0,"num_object_copies":14820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9054,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1231}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8780","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.645046+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:11.645046+0000","last_peered":"2024-09-15T21:10:11.645046+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:11.645046+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8780,"log_dups_size":0,"ondisk_log_size":8780,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30075416,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1340}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8890","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.705406+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:13.705406+0000","last_peered":"2024-09-15T21:10:13.705406+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:13.705406+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30078220,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8682","reported_seq":13346,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:11.767747+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:11.767747+0000","last_peered":"2024-09-15T21:10:11.767747+0000","last_clean":"2024-09-15T21:10:11.767747+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:11.767747+0000","last_undegraded":"2024-09-15T21:10:11.767747+0000","last_fullsized":"2024-09-15T21:10:11.767747+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8682,"log_dups_size":0,"ondisk_log_size":8682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30130176,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8685","reported_seq":8726,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.704002+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:11.704002+0000","last_peered":"2024-09-15T21:10:11.704002+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:11.704002+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8685,"log_dups_size":0,"ondisk_log_size":8685,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30653062,"num_objects":3742,"num_object_clones":0,"num_object_copies":14968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1486}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8792","reported_seq":8873,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.556653+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:13.556653+0000","last_peered":"2024-09-15T21:10:13.556653+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:13.556653+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30382726,"num_objects":3709,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8724","reported_seq":8782,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:11.669197+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:11.669197+0000","last_peered":"2024-09-15T21:10:11.669197+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:11.669197+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8724,"log_dups_size":0,"ondisk_log_size":8724,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30373132,"num_objects":3708,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8722,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1429}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.408080+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:13.408080+0000","last_peered":"2024-09-15T21:10:13.408080+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:13.408080+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077644,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8836","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:10.841287+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:10.841287+0000","last_peered":"2024-09-15T21:10:10.841287+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:10.841287+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8836,"log_dups_size":0,"ondisk_log_size":8836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30414092,"num_objects":3713,"num_object_clones":0,"num_object_copies":14852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":486709038,"num_objects":59418,"num_object_clones":0,"num_object_copies":237672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32884,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134994,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":971612160,"data_stored":971612160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140907,"ondisk_log_size":140907,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703204,"kb_used_data":238356,"kb_used_omap":4,"kb_used_meta":464827,"kb_avail":93668636,"statfs":{"total":96636764160,"available":95916683264,"internally_reserved":0,"allocated":244076544,"data_stored":243911555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":475983568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705100,"kb_used_data":236600,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93666740,"statfs":{"total":96636764160,"available":95914741760,"internally_reserved":0,"allocated":242278400,"data_stored":242113411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557236,"kb_used_data":236832,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93814604,"statfs":{"total":96636764160,"available":96066154496,"internally_reserved":0,"allocated":242515968,"data_stored":242350979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247897,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708272,"kb_used_data":239320,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93663568,"statfs":{"total":96636764160,"available":95911493632,"internally_reserved":0,"allocated":245063680,"data_stored":244898691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":480177870},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,8,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":244035584,"data_stored":244035584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242208768,"data_stored":242208768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241868800,"data_stored":241868800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":243499008,"data_stored":243499008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:18.007 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8668", 'reported_seq': 8715, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.601949+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:11.601949+0000', 'last_peered': '2024-09-15T21:10:11.601949+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:11.601949+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8668, 'log_dups_size': 0, 'ondisk_log_size': 8668, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30179328, 'num_objects': 3684, 'num_object_clones': 0, 'num_object_copies': 14736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8668, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1408}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'8874", 'reported_seq': 12822, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.093196+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:11.093196+0000', 'last_peered': '2024-09-15T21:10:11.093196+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:11.093196+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8874, 'log_dups_size': 0, 'ondisk_log_size': 8874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30127372, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 443, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8872, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1941, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3235}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8699", 'reported_seq': 8780, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:13.706694+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:13.706694+0000', 'last_peered': '2024-09-15T21:10:13.706694+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:13.706694+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8699, 'log_dups_size': 0, 'ondisk_log_size': 8699, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30579358, 'num_objects': 3734, 'num_object_clones': 0, 'num_object_copies': 14936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3734, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8698, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1476}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8783", 'reported_seq': 8853, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.067072+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:11.067072+0000', 'last_peered': '2024-09-15T21:10:11.067072+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:11.067072+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8783, 'log_dups_size': 0, 'ondisk_log_size': 8783, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30086412, 'num_objects': 3673, 'num_object_clones': 0, 'num_object_copies': 14692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3673, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8781, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:18.008 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'8875", 'reported_seq': 8931, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.732197+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:11.732197+0000', 'last_peered': '2024-09-15T21:10:11.732197+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:11.732197+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8875, 'log_dups_size': 0, 'ondisk_log_size': 8875, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30995724, 'num_objects': 3784, 'num_object_clones': 0, 'num_object_copies': 15136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3784, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8873, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1473}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8721", 'reported_seq': 8767, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.642269+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:11.642269+0000', 'last_peered': '2024-09-15T21:10:11.642269+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:11.642269+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8721, 'log_dups_size': 0, 'ondisk_log_size': 8721, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29941760, 'num_objects': 3655, 'num_object_clones': 0, 'num_object_copies': 14620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8721, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1339}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'8955", 'reported_seq': 9026, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.809038+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:10.809038+0000', 'last_peered': '2024-09-15T21:10:10.809038+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:10.809038+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8955, 'log_dups_size': 0, 'ondisk_log_size': 8955, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31266060, 'num_objects': 3817, 'num_object_clones': 0, 'num_object_copies': 15268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8953, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1451}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9056", 'reported_seq': 9126, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.851975+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:10.851975+0000', 'last_peered': '2024-09-15T21:10:10.851975+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:10.851975+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9056, 'log_dups_size': 0, 'ondisk_log_size': 9056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30348556, 'num_objects': 3705, 'num_object_clones': 0, 'num_object_copies': 14820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9054, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1231}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8780", 'reported_seq': 8841, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.645046+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:11.645046+0000', 'last_peered': '2024-09-15T21:10:11.645046+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:11.645046+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8780, 'log_dups_size': 0, 'ondisk_log_size': 8780, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30075416, 'num_objects': 3672, 'num_object_clones': 0, 'num_object_copies': 14688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8777, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8890", 'reported_seq': 8971, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:13.705406+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:13.705406+0000', 'last_peered': '2024-09-15T21:10:13.705406+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:13.705406+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8890, 'log_dups_size': 0, 'ondisk_log_size': 8890, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30078220, 'num_objects': 3672, 'num_object_clones': 0, 'num_object_copies': 14688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3672, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8888, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1337}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:18.009 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8685", 'reported_seq': 8726, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.704002+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:11.704002+0000', 'last_peered': '2024-09-15T21:10:11.704002+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:11.704002+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8685, 'log_dups_size': 0, 'ondisk_log_size': 8685, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30653062, 'num_objects': 3742, 'num_object_clones': 0, 'num_object_copies': 14968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8684, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8792", 'reported_seq': 8873, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:13.556653+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:13.556653+0000', 'last_peered': '2024-09-15T21:10:13.556653+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:13.556653+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8792, 'log_dups_size': 0, 'ondisk_log_size': 8792, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30382726, 'num_objects': 3709, 'num_object_clones': 0, 'num_object_copies': 14836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3709, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8791, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1423}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8724", 'reported_seq': 8782, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:11.669197+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:11.669197+0000', 'last_peered': '2024-09-15T21:10:11.669197+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:11.669197+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8724, 'log_dups_size': 0, 'ondisk_log_size': 8724, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30373132, 'num_objects': 3708, 'num_object_clones': 0, 'num_object_copies': 14832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3708, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8722, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8887", 'reported_seq': 8971, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:13.408080+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:13.408080+0000', 'last_peered': '2024-09-15T21:10:13.408080+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:13.408080+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8887, 'log_dups_size': 0, 'ondisk_log_size': 8887, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 31077644, 'num_objects': 3794, 'num_object_clones': 0, 'num_object_copies': 15176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8885, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:18.010 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8836", 'reported_seq': 8906, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:10.841287+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:10.841287+0000', 'last_peered': '2024-09-15T21:10:10.841287+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:10.841287+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8836, 'log_dups_size': 0, 'ondisk_log_size': 8836, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30414092, 'num_objects': 3713, 'num_object_clones': 0, 'num_object_copies': 14852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8834, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1398}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:18.011 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:18.723 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:18.723 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32807/234284 objects degraded (14.003%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":484,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":58571,"data_bytes":479770414,"bytes_used":2737680384,"bytes_avail":383809376256,"bytes_total":386547056640,"degraded_objects":32807,"degraded_total":234284,"degraded_ratio":0.14003090266514145,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:18.753 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:18.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:19.370 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:19.370 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:19.403 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-15T21:10:18.072090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":479770414,"num_objects":58571,"num_object_clones":0,"num_object_copies":234284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135841,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2040,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141754,"ondisk_log_size":141754,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2673516,"kb_used_data":942496,"kb_used_omap":14,"kb_used_meta":1730929,"kb_avail":374813844,"statfs":{"total":386547056640,"available":383809376256,"internally_reserved":0,"allocated":965115904,"data_stored":964455948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1772471858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,12,2,7,2,11,18,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21979136,"num_objects":-2683,"num_object_clones":0,"num_object_copies":-10732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2683,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004740"},"pg_stats":[{"pgid":"1.f","version":"33'8729","reported_seq":8776,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.656188+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:16.656188+0000","last_peered":"2024-09-15T21:10:16.656188+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:16.656188+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8729,"log_dups_size":0,"ondisk_log_size":8729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29679616,"num_objects":3623,"num_object_clones":0,"num_object_copies":14492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8729,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'8939","reported_seq":13039,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.124000+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:16.124000+0000","last_peered":"2024-09-15T21:10:16.124000+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:16.124000+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8939,"log_dups_size":0,"ondisk_log_size":8939,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29594892,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2018,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3247}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8699","reported_seq":8780,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.706694+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:13.706694+0000","last_peered":"2024-09-15T21:10:13.706694+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:13.706694+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8699,"log_dups_size":0,"ondisk_log_size":8699,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30579358,"num_objects":3734,"num_object_clones":0,"num_object_copies":14936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8698,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1476}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8852","reported_seq":8922,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:15.997549+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:15.997549+0000","last_peered":"2024-09-15T21:10:15.997549+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:15.997549+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8852,"log_dups_size":0,"ondisk_log_size":8852,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29521164,"num_objects":3604,"num_object_clones":0,"num_object_copies":14416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1270}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'8963","reported_seq":9020,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.805019+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:16.805019+0000","last_peered":"2024-09-15T21:10:16.805019+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:16.805019+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8963,"log_dups_size":0,"ondisk_log_size":8963,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30274828,"num_objects":3696,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1385}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8795","reported_seq":8841,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.696202+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:16.696202+0000","last_peered":"2024-09-15T21:10:16.696202+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:16.696202+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8795,"log_dups_size":0,"ondisk_log_size":8795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29335552,"num_objects":3581,"num_object_clones":0,"num_object_copies":14324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8795,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9022","reported_seq":9093,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:15.987129+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:15.987129+0000","last_peered":"2024-09-15T21:10:15.987129+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:15.987129+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30717196,"num_objects":3750,"num_object_clones":0,"num_object_copies":15000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1384}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9113","reported_seq":9182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.100317+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:16.100317+0000","last_peered":"2024-09-15T21:10:16.100317+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:16.100317+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9113,"log_dups_size":0,"ondisk_log_size":9113,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29881612,"num_objects":3648,"num_object_clones":0,"num_object_copies":14592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9111,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1174}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8847","reported_seq":8908,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.778993+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:16.778993+0000","last_peered":"2024-09-15T21:10:16.778993+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:16.778993+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8847,"log_dups_size":0,"ondisk_log_size":8847,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29526552,"num_objects":3605,"num_object_clones":0,"num_object_copies":14420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8844,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1273}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8890","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.705406+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:13.705406+0000","last_peered":"2024-09-15T21:10:13.705406+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:13.705406+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30078220,"num_objects":3672,"num_object_clones":0,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1337}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8738","reported_seq":13402,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:16.636788+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:16.636788+0000","last_peered":"2024-09-15T21:10:16.636788+0000","last_clean":"2024-09-15T21:10:16.636788+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:16.636788+0000","last_undegraded":"2024-09-15T21:10:16.636788+0000","last_fullsized":"2024-09-15T21:10:16.636788+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8738,"log_dups_size":0,"ondisk_log_size":8738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29671424,"num_objects":3622,"num_object_clones":0,"num_object_copies":14488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2849,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8762","reported_seq":8803,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.742077+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:16.742077+0000","last_peered":"2024-09-15T21:10:16.742077+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:16.742077+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8762,"log_dups_size":0,"ondisk_log_size":8762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30022278,"num_objects":3665,"num_object_clones":0,"num_object_copies":14660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8761,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1409}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8792","reported_seq":8873,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.556653+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:13.556653+0000","last_peered":"2024-09-15T21:10:13.556653+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:13.556653+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8792,"log_dups_size":0,"ondisk_log_size":8792,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30382726,"num_objects":3709,"num_object_clones":0,"num_object_copies":14836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8791,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1423}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8816","reported_seq":8874,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.812752+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:16.812752+0000","last_peered":"2024-09-15T21:10:16.812752+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:16.812752+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8816,"log_dups_size":0,"ondisk_log_size":8816,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29619468,"num_objects":3616,"num_object_clones":0,"num_object_copies":14464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8814,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1337}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8887","reported_seq":8971,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:13.408080+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:13.408080+0000","last_peered":"2024-09-15T21:10:13.408080+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:13.408080+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8887,"log_dups_size":0,"ondisk_log_size":8887,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31077644,"num_objects":3794,"num_object_clones":0,"num_object_copies":15176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1513}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8910","reported_seq":8978,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:16.101740+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:16.101740+0000","last_peered":"2024-09-15T21:10:16.101740+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:16.101740+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8910,"log_dups_size":0,"ondisk_log_size":8910,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29807884,"num_objects":3639,"num_object_clones":0,"num_object_copies":14556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8908,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":479770414,"num_objects":58571,"num_object_clones":0,"num_object_copies":234284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58571,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135841,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2040,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":962789376,"data_stored":962789376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141754,"ondisk_log_size":141754,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703092,"kb_used_data":234084,"kb_used_omap":4,"kb_used_meta":468987,"kb_avail":93668748,"statfs":{"total":96636764160,"available":95916797952,"internally_reserved":0,"allocated":239702016,"data_stored":239537027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":480243408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,0,4,1,2,10,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705100,"kb_used_data":236600,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93666740,"statfs":{"total":96636764160,"available":95914741760,"internally_reserved":0,"allocated":242278400,"data_stored":242113411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557236,"kb_used_data":236832,"kb_used_omap":4,"kb_used_meta":320379,"kb_avail":93814604,"statfs":{"total":96636764160,"available":96066154496,"internally_reserved":0,"allocated":242515968,"data_stored":242350979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":328068815},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,0,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247898,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708088,"kb_used_data":234980,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93663752,"statfs":{"total":96636764160,"available":95911682048,"internally_reserved":0,"allocated":240619520,"data_stored":240454531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":484437710},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1,0,4,4,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":239706112,"data_stored":239706112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":242208768,"data_stored":242208768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241868800,"data_stored":241868800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":239005696,"data_stored":239005696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:22.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:23.010 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:23.010 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:23.032 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-15T21:10:22.076937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":471078702,"num_objects":57510,"num_object_clones":0,"num_object_copies":230040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136902,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142815,"ondisk_log_size":142815,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710600,"kb_used_data":925432,"kb_used_omap":14,"kb_used_meta":1785073,"kb_avail":374776760,"statfs":{"total":386547056640,"available":383771402240,"internally_reserved":0,"allocated":947642368,"data_stored":946982412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1827915314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,8,11,24,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24395776,"num_objects":-2978,"num_object_clones":0,"num_object_copies":-11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2978,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008204"},"pg_stats":[{"pgid":"1.f","version":"33'8797","reported_seq":8844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.781977+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:21.781977+0000","last_peered":"2024-09-15T21:10:21.781977+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:21.781977+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9019","reported_seq":13261,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.156003+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:21.156003+0000","last_peered":"2024-09-15T21:10:21.156003+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:21.156003+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28939532,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2090,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8755","reported_seq":8836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.602390+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:18.602390+0000","last_peered":"2024-09-15T21:10:18.602390+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:18.602390+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30120606,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8754,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1420}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8912","reported_seq":8982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.970042+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:20.970042+0000","last_peered":"2024-09-15T21:10:20.970042+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:20.970042+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029644,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9045","reported_seq":9102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839110+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:21.839110+0000","last_peered":"2024-09-15T21:10:21.839110+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:21.839110+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603084,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8864","reported_seq":8910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:21.839643+0000","last_peered":"2024-09-15T21:10:21.839643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:21.839643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9083","reported_seq":9154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.798245+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:20.798245+0000","last_peered":"2024-09-15T21:10:20.798245+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:20.798245+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217484,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9183","reported_seq":9253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.088007+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:21.088007+0000","last_peered":"2024-09-15T21:10:21.088007+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:21.088007+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308172,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8914","reported_seq":8975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.714760+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:21.714760+0000","last_peered":"2024-09-15T21:10:21.714760+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:21.714760+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28977688,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8954","reported_seq":9036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.696856+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:18.696856+0000","last_peered":"2024-09-15T21:10:18.696856+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:18.696856+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8954,"log_dups_size":0,"ondisk_log_size":8954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29553932,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8799","reported_seq":13463,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:21.612132+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:21.612132+0000","last_peered":"2024-09-15T21:10:21.612132+0000","last_clean":"2024-09-15T21:10:21.612132+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:21.612132+0000","last_undegraded":"2024-09-15T21:10:21.612132+0000","last_fullsized":"2024-09-15T21:10:21.612132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8799,"log_dups_size":0,"ondisk_log_size":8799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29171712,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8831","reported_seq":8870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.812541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:21.812541+0000","last_peered":"2024-09-15T21:10:21.812541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:21.812541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8831,"log_dups_size":0,"ondisk_log_size":8831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29457030,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8864","reported_seq":8945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.697485+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:18.697485+0000","last_peered":"2024-09-15T21:10:18.697485+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:18.697485+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29792902,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8874","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.704786+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:21.704786+0000","last_peered":"2024-09-15T21:10:21.704786+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:21.704786+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144332,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8952","reported_seq":9036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.553762+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:18.553762+0000","last_peered":"2024-09-15T21:10:18.553762+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:18.553762+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8952,"log_dups_size":0,"ondisk_log_size":8952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30545164,"num_objects":3729,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1448}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8969","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.941248+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:20.941248+0000","last_peered":"2024-09-15T21:10:20.941248+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:20.941248+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29324556,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":471078702,"num_objects":57510,"num_object_clones":0,"num_object_copies":230040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136902,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945057792,"data_stored":945057792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142815,"ondisk_log_size":142815,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721016,"kb_used_data":229800,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93650824,"statfs":{"total":96636764160,"available":95898443776,"internally_reserved":0,"allocated":235315200,"data_stored":235150211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722988,"kb_used_data":232280,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93648852,"statfs":{"total":96636764160,"available":95896424448,"internally_reserved":0,"allocated":237854720,"data_stored":237689731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557140,"kb_used_data":232568,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93814700,"statfs":{"total":96636764160,"available":96066252800,"internally_reserved":0,"allocated":238149632,"data_stored":237984643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709456,"kb_used_data":230784,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93662384,"statfs":{"total":96636764160,"available":95910281216,"internally_reserved":0,"allocated":236322816,"data_stored":236157827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":490139342},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235278336,"data_stored":235278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237760512,"data_stored":237760512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237436928,"data_stored":237436928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234582016,"data_stored":234582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:23.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:23.652 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:23.652 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:23.686 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-15T21:10:22.076937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":471078702,"num_objects":57510,"num_object_clones":0,"num_object_copies":230040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136902,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142815,"ondisk_log_size":142815,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2710600,"kb_used_data":925432,"kb_used_omap":14,"kb_used_meta":1785073,"kb_avail":374776760,"statfs":{"total":386547056640,"available":383771402240,"internally_reserved":0,"allocated":947642368,"data_stored":946982412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1827915314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,8,11,24,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24395776,"num_objects":-2978,"num_object_clones":0,"num_object_copies":-11912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2978,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":234,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008204"},"pg_stats":[{"pgid":"1.f","version":"33'8797","reported_seq":8844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.781977+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:21.781977+0000","last_peered":"2024-09-15T21:10:21.781977+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:21.781977+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9019","reported_seq":13261,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.156003+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:21.156003+0000","last_peered":"2024-09-15T21:10:21.156003+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:21.156003+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28939532,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2090,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8755","reported_seq":8836,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.602390+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:18.602390+0000","last_peered":"2024-09-15T21:10:18.602390+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:18.602390+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8755,"log_dups_size":0,"ondisk_log_size":8755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30120606,"num_objects":3678,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3678,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8754,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1420}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8912","reported_seq":8982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.970042+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:20.970042+0000","last_peered":"2024-09-15T21:10:20.970042+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:20.970042+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029644,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9045","reported_seq":9102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839110+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:21.839110+0000","last_peered":"2024-09-15T21:10:21.839110+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:21.839110+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603084,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8864","reported_seq":8910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:21.839643+0000","last_peered":"2024-09-15T21:10:21.839643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:21.839643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9083","reported_seq":9154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.798245+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:20.798245+0000","last_peered":"2024-09-15T21:10:20.798245+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:20.798245+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217484,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9183","reported_seq":9253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.088007+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:21.088007+0000","last_peered":"2024-09-15T21:10:21.088007+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:21.088007+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308172,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8914","reported_seq":8975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.714760+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:21.714760+0000","last_peered":"2024-09-15T21:10:21.714760+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:21.714760+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28977688,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'8954","reported_seq":9036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.696856+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:18.696856+0000","last_peered":"2024-09-15T21:10:18.696856+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:18.696856+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8954,"log_dups_size":0,"ondisk_log_size":8954,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29553932,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1273}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8799","reported_seq":13463,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:21.612132+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:21.612132+0000","last_peered":"2024-09-15T21:10:21.612132+0000","last_clean":"2024-09-15T21:10:21.612132+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:21.612132+0000","last_undegraded":"2024-09-15T21:10:21.612132+0000","last_fullsized":"2024-09-15T21:10:21.612132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8799,"log_dups_size":0,"ondisk_log_size":8799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29171712,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8831","reported_seq":8870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.812541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:21.812541+0000","last_peered":"2024-09-15T21:10:21.812541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:21.812541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8831,"log_dups_size":0,"ondisk_log_size":8831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29457030,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8864","reported_seq":8945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.697485+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:18.697485+0000","last_peered":"2024-09-15T21:10:18.697485+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:18.697485+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29792902,"num_objects":3637,"num_object_clones":0,"num_object_copies":14548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8863,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1351}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8874","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.704786+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:21.704786+0000","last_peered":"2024-09-15T21:10:21.704786+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:21.704786+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144332,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'8952","reported_seq":9036,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:18.553762+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:18.553762+0000","last_peered":"2024-09-15T21:10:18.553762+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:18.553762+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8952,"log_dups_size":0,"ondisk_log_size":8952,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30545164,"num_objects":3729,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8950,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1448}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8969","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.941248+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:20.941248+0000","last_peered":"2024-09-15T21:10:20.941248+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:20.941248+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29324556,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":471078702,"num_objects":57510,"num_object_clones":0,"num_object_copies":230040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57510,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136902,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":945057792,"data_stored":945057792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":142815,"ondisk_log_size":142815,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721016,"kb_used_data":229800,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93650824,"statfs":{"total":96636764160,"available":95898443776,"internally_reserved":0,"allocated":235315200,"data_stored":235150211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722988,"kb_used_data":232280,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93648852,"statfs":{"total":96636764160,"available":95896424448,"internally_reserved":0,"allocated":237854720,"data_stored":237689731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557140,"kb_used_data":232568,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93814700,"statfs":{"total":96636764160,"available":96066252800,"internally_reserved":0,"allocated":238149632,"data_stored":237984643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709456,"kb_used_data":230784,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93662384,"statfs":{"total":96636764160,"available":95910281216,"internally_reserved":0,"allocated":236322816,"data_stored":236157827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":490139342},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235278336,"data_stored":235278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237760512,"data_stored":237760512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":237436928,"data_stored":237436928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234582016,"data_stored":234582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8797", 'reported_seq': 8844, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.781977+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:21.781977+0000', 'last_peered': '2024-09-15T21:10:21.781977+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:21.781977+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8797, 'log_dups_size': 0, 'ondisk_log_size': 8797, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29122560, 'num_objects': 3555, 'num_object_clones': 0, 'num_object_copies': 14220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3555, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8797, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1279}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'9019", 'reported_seq': 13261, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.156003+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:21.156003+0000', 'last_peered': '2024-09-15T21:10:21.156003+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:21.156003+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9019, 'log_dups_size': 0, 'ondisk_log_size': 9019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28939532, 'num_objects': 3533, 'num_object_clones': 0, 'num_object_copies': 14132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 294, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9017, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2090, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8755", 'reported_seq': 8836, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:18.602390+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:18.602390+0000', 'last_peered': '2024-09-15T21:10:18.602390+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:18.602390+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8755, 'log_dups_size': 0, 'ondisk_log_size': 8755, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30120606, 'num_objects': 3678, 'num_object_clones': 0, 'num_object_copies': 14712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3678, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8754, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1420}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:23.687 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8912", 'reported_seq': 8982, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:20.970042+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:20.970042+0000', 'last_peered': '2024-09-15T21:10:20.970042+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:20.970042+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8912, 'log_dups_size': 0, 'ondisk_log_size': 8912, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29029644, 'num_objects': 3544, 'num_object_clones': 0, 'num_object_copies': 14176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3544, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8910, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'9045", 'reported_seq': 9102, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.839110+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:21.839110+0000', 'last_peered': '2024-09-15T21:10:21.839110+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:21.839110+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9045, 'log_dups_size': 0, 'ondisk_log_size': 9045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29603084, 'num_objects': 3614, 'num_object_clones': 0, 'num_object_copies': 14456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9043, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1303}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8864", 'reported_seq': 8910, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.839643+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:21.839643+0000', 'last_peered': '2024-09-15T21:10:21.839643+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:21.839643+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8864, 'log_dups_size': 0, 'ondisk_log_size': 8864, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28770304, 'num_objects': 3512, 'num_object_clones': 0, 'num_object_copies': 14048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8864, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9083", 'reported_seq': 9154, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:20.798245+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:20.798245+0000', 'last_peered': '2024-09-15T21:10:20.798245+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:20.798245+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9083, 'log_dups_size': 0, 'ondisk_log_size': 9083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30217484, 'num_objects': 3689, 'num_object_clones': 0, 'num_object_copies': 14756, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3689, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9081, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1323}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9183", 'reported_seq': 9253, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.088007+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:21.088007+0000', 'last_peered': '2024-09-15T21:10:21.088007+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:21.088007+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9183, 'log_dups_size': 0, 'ondisk_log_size': 9183, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29308172, 'num_objects': 3578, 'num_object_clones': 0, 'num_object_copies': 14312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9181, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1104}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:23.688 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8914", 'reported_seq': 8975, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.714760+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:21.714760+0000', 'last_peered': '2024-09-15T21:10:21.714760+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:21.714760+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8914, 'log_dups_size': 0, 'ondisk_log_size': 8914, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28977688, 'num_objects': 3538, 'num_object_clones': 0, 'num_object_copies': 14152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3538, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8911, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1206}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'8954", 'reported_seq': 9036, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:18.696856+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:18.696856+0000', 'last_peered': '2024-09-15T21:10:18.696856+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:18.696856+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8954, 'log_dups_size': 0, 'ondisk_log_size': 8954, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29553932, 'num_objects': 3608, 'num_object_clones': 0, 'num_object_copies': 14432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8952, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1273}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8831", 'reported_seq': 8870, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.812541+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:21.812541+0000', 'last_peered': '2024-09-15T21:10:21.812541+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:21.812541+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8831, 'log_dups_size': 0, 'ondisk_log_size': 8831, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29457030, 'num_objects': 3596, 'num_object_clones': 0, 'num_object_copies': 14384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8830, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1340}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8864", 'reported_seq': 8945, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:18.697485+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:18.697485+0000', 'last_peered': '2024-09-15T21:10:18.697485+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:18.697485+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8864, 'log_dups_size': 0, 'ondisk_log_size': 8864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29792902, 'num_objects': 3637, 'num_object_clones': 0, 'num_object_copies': 14548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3637, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8863, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1351}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8874", 'reported_seq': 8932, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:21.704786+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:21.704786+0000', 'last_peered': '2024-09-15T21:10:21.704786+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:21.704786+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8874, 'log_dups_size': 0, 'ondisk_log_size': 8874, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29144332, 'num_objects': 3558, 'num_object_clones': 0, 'num_object_copies': 14232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3558, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8872, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:23.689 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'8952", 'reported_seq': 9036, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:18.553762+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:18.553762+0000', 'last_peered': '2024-09-15T21:10:18.553762+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:18.553762+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8952, 'log_dups_size': 0, 'ondisk_log_size': 8952, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30545164, 'num_objects': 3729, 'num_object_clones': 0, 'num_object_copies': 14916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8950, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1448}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:23.690 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'8969", 'reported_seq': 9039, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:20.941248+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:20.941248+0000', 'last_peered': '2024-09-15T21:10:20.941248+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:20.941248+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8969, 'log_dups_size': 0, 'ondisk_log_size': 8969, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29324556, 'num_objects': 3580, 'num_object_clones': 0, 'num_object_copies': 14320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3580, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8967, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1265}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:23.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:24.457 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:24.457 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32735/228892 objects degraded (14.302%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":490,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":57223,"data_bytes":468727598,"bytes_used":2758352896,"bytes_avail":383788703744,"bytes_total":386547056640,"degraded_objects":32735,"degraded_total":228892,"degraded_ratio":0.1430150463974276,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:24.479 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:24.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:25.084 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:25.085 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:25.118 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2024-09-15T21:10:24.077794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":468727598,"num_objects":57223,"num_object_clones":0,"num_object_copies":228892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137189,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143102,"ondisk_log_size":143102,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2693704,"kb_used_data":921008,"kb_used_omap":14,"kb_used_meta":1772593,"kb_avail":374793656,"statfs":{"total":386547056640,"available":383788703744,"internally_reserved":0,"allocated":943112192,"data_stored":942452236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1815135794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,9,9,11,26,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20021248,"num_objects":-2444,"num_object_clones":0,"num_object_copies":-9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008405"},"pg_stats":[{"pgid":"1.f","version":"33'8797","reported_seq":8844,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.781977+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:21.781977+0000","last_peered":"2024-09-15T21:10:21.781977+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:21.781977+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8797,"log_dups_size":0,"ondisk_log_size":8797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29122560,"num_objects":3555,"num_object_clones":0,"num_object_copies":14220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8797,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1279}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9019","reported_seq":13261,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.156003+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:21.156003+0000","last_peered":"2024-09-15T21:10:21.156003+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:21.156003+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9019,"log_dups_size":0,"ondisk_log_size":9019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28939532,"num_objects":3533,"num_object_clones":0,"num_object_copies":14132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2090,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3239}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8825","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.775268+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:23.775268+0000","last_peered":"2024-09-15T21:10:23.775268+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:23.775268+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29547166,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8824,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8912","reported_seq":8982,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.970042+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:20.970042+0000","last_peered":"2024-09-15T21:10:20.970042+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:20.970042+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8912,"log_dups_size":0,"ondisk_log_size":8912,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29029644,"num_objects":3544,"num_object_clones":0,"num_object_copies":14176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8910,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9045","reported_seq":9102,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839110+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:21.839110+0000","last_peered":"2024-09-15T21:10:21.839110+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:21.839110+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9045,"log_dups_size":0,"ondisk_log_size":9045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29603084,"num_objects":3614,"num_object_clones":0,"num_object_copies":14456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8864","reported_seq":8910,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.839643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:21.839643+0000","last_peered":"2024-09-15T21:10:21.839643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:21.839643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8864,"log_dups_size":0,"ondisk_log_size":8864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28770304,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9083","reported_seq":9154,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.798245+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:20.798245+0000","last_peered":"2024-09-15T21:10:20.798245+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:20.798245+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9083,"log_dups_size":0,"ondisk_log_size":9083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30217484,"num_objects":3689,"num_object_clones":0,"num_object_copies":14756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9081,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1323}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9183","reported_seq":9253,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.088007+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:21.088007+0000","last_peered":"2024-09-15T21:10:21.088007+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:21.088007+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9183,"log_dups_size":0,"ondisk_log_size":9183,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29308172,"num_objects":3578,"num_object_clones":0,"num_object_copies":14312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9181,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1104}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8914","reported_seq":8975,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.714760+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:21.714760+0000","last_peered":"2024-09-15T21:10:21.714760+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:21.714760+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8914,"log_dups_size":0,"ondisk_log_size":8914,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28977688,"num_objects":3538,"num_object_clones":0,"num_object_copies":14152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8911,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1206}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9022","reported_seq":9104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.649892+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:23.649892+0000","last_peered":"2024-09-15T21:10:23.649892+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:23.649892+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8799","reported_seq":13463,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:21.612132+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:21.612132+0000","last_peered":"2024-09-15T21:10:21.612132+0000","last_clean":"2024-09-15T21:10:21.612132+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:21.612132+0000","last_undegraded":"2024-09-15T21:10:21.612132+0000","last_fullsized":"2024-09-15T21:10:21.612132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8799,"log_dups_size":0,"ondisk_log_size":8799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29171712,"num_objects":3561,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2910,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8831","reported_seq":8870,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.812541+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:21.812541+0000","last_peered":"2024-09-15T21:10:21.812541+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:21.812541+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8831,"log_dups_size":0,"ondisk_log_size":8831,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29457030,"num_objects":3596,"num_object_clones":0,"num_object_copies":14384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1340}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8941","reported_seq":9022,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.630939+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:23.630939+0000","last_peered":"2024-09-15T21:10:23.630939+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:23.630939+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8874","reported_seq":8932,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:21.704786+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:21.704786+0000","last_peered":"2024-09-15T21:10:21.704786+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:21.704786+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8874,"log_dups_size":0,"ondisk_log_size":8874,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29144332,"num_objects":3558,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8872,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9024","reported_seq":9108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.610859+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:23.610859+0000","last_peered":"2024-09-15T21:10:23.610859+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:23.610859+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955340,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'8969","reported_seq":9039,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:20.941248+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:20.941248+0000","last_peered":"2024-09-15T21:10:20.941248+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:20.941248+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29324556,"num_objects":3580,"num_object_clones":0,"num_object_copies":14320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8967,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1265}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":468727598,"num_objects":57223,"num_object_clones":0,"num_object_copies":228892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32735,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57223,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137189,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":940601344,"data_stored":940601344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143102,"ondisk_log_size":143102,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721016,"kb_used_data":229800,"kb_used_omap":4,"kb_used_meta":491195,"kb_avail":93650824,"statfs":{"total":96636764160,"available":95898443776,"internally_reserved":0,"allocated":235315200,"data_stored":235150211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":502984400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706092,"kb_used_data":227856,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93665748,"statfs":{"total":96636764160,"available":95913725952,"internally_reserved":0,"allocated":233324544,"data_stored":233159555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051661,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":557140,"kb_used_data":232568,"kb_used_omap":4,"kb_used_meta":324539,"kb_avail":93814700,"statfs":{"total":96636764160,"available":96066252800,"internally_reserved":0,"allocated":238149632,"data_stored":237984643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":332328655},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247899,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709456,"kb_used_data":230784,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93662384,"statfs":{"total":96636764160,"available":95910281216,"internally_reserved":0,"allocated":236322816,"data_stored":236157827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":490139342},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,5,13,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":235278336,"data_stored":235278336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":237760512,"data_stored":237760512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232980480,"data_stored":232980480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":234582016,"data_stored":234582016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:28.120 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:28.773 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:28.774 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:28.796 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-15T21:10:28.080091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462288686,"num_objects":56437,"num_object_clones":0,"num_object_copies":225748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137975,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143888,"ondisk_log_size":143888,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2711432,"kb_used_data":908216,"kb_used_omap":14,"kb_used_meta":1803121,"kb_avail":374775928,"statfs":{"total":386547056640,"available":383770550272,"internally_reserved":0,"allocated":930013184,"data_stored":929353228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1846396466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,9,7,14,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24420352,"num_objects":-2981,"num_object_clones":0,"num_object_copies":-11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008925"},"pg_stats":[{"pgid":"1.f","version":"33'8865","reported_seq":8912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.861866+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:26.861866+0000","last_peered":"2024-09-15T21:10:26.861866+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:26.861866+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28565504,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9097","reported_seq":13470,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.162316+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:26.162316+0000","last_peered":"2024-09-15T21:10:26.162316+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:26.162316+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300556,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8825","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.775268+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:23.775268+0000","last_peered":"2024-09-15T21:10:23.775268+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:23.775268+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29547166,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8824,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8985","reported_seq":9055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.057886+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:26.057886+0000","last_peered":"2024-09-15T21:10:26.057886+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:26.057886+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431628,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8983,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9112","reported_seq":9169,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.898465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:26.898465+0000","last_peered":"2024-09-15T21:10:26.898465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:26.898465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29054220,"num_objects":3547,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8930","reported_seq":8974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.858065+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:26.858065+0000","last_peered":"2024-09-15T21:10:26.858065+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:26.858065+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28229632,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8930,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9142","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.084526+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:26.084526+0000","last_peered":"2024-09-15T21:10:26.084526+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:26.084526+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734156,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9249","reported_seq":9319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.075143+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:26.075143+0000","last_peered":"2024-09-15T21:10:26.075143+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:26.075143+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28767500,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8984","reported_seq":9045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.687071+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:26.687071+0000","last_peered":"2024-09-15T21:10:26.687071+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:26.687071+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28404248,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9022","reported_seq":9104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.649892+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:23.649892+0000","last_peered":"2024-09-15T21:10:23.649892+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:23.649892+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8850","reported_seq":13514,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:26.749104+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:26.749104+0000","last_peered":"2024-09-15T21:10:26.749104+0000","last_clean":"2024-09-15T21:10:26.749104+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:26.749104+0000","last_undegraded":"2024-09-15T21:10:26.749104+0000","last_fullsized":"2024-09-15T21:10:26.749104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28753920,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8904","reported_seq":8945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.865076+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:26.865076+0000","last_peered":"2024-09-15T21:10:26.865076+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:26.865076+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28859014,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8941","reported_seq":9022,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.630939+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:23.630939+0000","last_peered":"2024-09-15T21:10:23.630939+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:23.630939+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8934","reported_seq":8992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.727315+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:26.727315+0000","last_peered":"2024-09-15T21:10:26.727315+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:26.727315+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652812,"num_objects":3498,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9024","reported_seq":9108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.610859+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:23.610859+0000","last_peered":"2024-09-15T21:10:23.610859+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:23.610859+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955340,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9024","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:25.916120+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:25.916120+0000","last_peered":"2024-09-15T21:10:25.916120+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:25.916120+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873996,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462288686,"num_objects":56437,"num_object_clones":0,"num_object_copies":225748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137975,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927363072,"data_stored":927363072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143888,"ondisk_log_size":143888,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721032,"kb_used_data":225656,"kb_used_omap":4,"kb_used_meta":495355,"kb_avail":93650808,"statfs":{"total":96636764160,"available":95898427392,"internally_reserved":0,"allocated":231071744,"data_stored":230906755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":507244240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706092,"kb_used_data":227856,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93665748,"statfs":{"total":96636764160,"available":95913725952,"internally_reserved":0,"allocated":233324544,"data_stored":233159555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574932,"kb_used_data":228164,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93796908,"statfs":{"total":96636764160,"available":96048033792,"internally_reserved":0,"allocated":233639936,"data_stored":233474947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":226540,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":231976960,"data_stored":231811971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231002112,"data_stored":231002112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233132032,"data_stored":233132032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232980480,"data_stored":232980480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230248448,"data_stored":230248448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:28.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:29.606 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:29.608 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:29.640 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-15T21:10:28.080091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462288686,"num_objects":56437,"num_object_clones":0,"num_object_copies":225748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137975,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143888,"ondisk_log_size":143888,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2711432,"kb_used_data":908216,"kb_used_omap":14,"kb_used_meta":1803121,"kb_avail":374775928,"statfs":{"total":386547056640,"available":383770550272,"internally_reserved":0,"allocated":930013184,"data_stored":929353228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1846396466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,7,9,7,14,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24420352,"num_objects":-2981,"num_object_clones":0,"num_object_copies":-11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2981,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008925"},"pg_stats":[{"pgid":"1.f","version":"33'8865","reported_seq":8912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.861866+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:26.861866+0000","last_peered":"2024-09-15T21:10:26.861866+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:26.861866+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28565504,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9097","reported_seq":13470,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.162316+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:26.162316+0000","last_peered":"2024-09-15T21:10:26.162316+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:26.162316+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300556,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8825","reported_seq":8906,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.775268+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:23.775268+0000","last_peered":"2024-09-15T21:10:23.775268+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:23.775268+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8825,"log_dups_size":0,"ondisk_log_size":8825,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29547166,"num_objects":3608,"num_object_clones":0,"num_object_copies":14432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8824,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1350}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8985","reported_seq":9055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.057886+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:26.057886+0000","last_peered":"2024-09-15T21:10:26.057886+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:26.057886+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431628,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8983,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9112","reported_seq":9169,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.898465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:26.898465+0000","last_peered":"2024-09-15T21:10:26.898465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:26.898465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29054220,"num_objects":3547,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8930","reported_seq":8974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.858065+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:26.858065+0000","last_peered":"2024-09-15T21:10:26.858065+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:26.858065+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28229632,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8930,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9142","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.084526+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:26.084526+0000","last_peered":"2024-09-15T21:10:26.084526+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:26.084526+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734156,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9249","reported_seq":9319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.075143+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:26.075143+0000","last_peered":"2024-09-15T21:10:26.075143+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:26.075143+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28767500,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8984","reported_seq":9045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.687071+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:26.687071+0000","last_peered":"2024-09-15T21:10:26.687071+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:26.687071+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28404248,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9022","reported_seq":9104,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.649892+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:23.649892+0000","last_peered":"2024-09-15T21:10:23.649892+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:23.649892+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9022,"log_dups_size":0,"ondisk_log_size":9022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28996876,"num_objects":3540,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9020,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1205}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8850","reported_seq":13514,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:26.749104+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:26.749104+0000","last_peered":"2024-09-15T21:10:26.749104+0000","last_clean":"2024-09-15T21:10:26.749104+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:26.749104+0000","last_undegraded":"2024-09-15T21:10:26.749104+0000","last_fullsized":"2024-09-15T21:10:26.749104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28753920,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8904","reported_seq":8945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.865076+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:26.865076+0000","last_peered":"2024-09-15T21:10:26.865076+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:26.865076+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28859014,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8941","reported_seq":9022,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.630939+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:23.630939+0000","last_peered":"2024-09-15T21:10:23.630939+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:23.630939+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8941,"log_dups_size":0,"ondisk_log_size":8941,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29162118,"num_objects":3560,"num_object_clones":0,"num_object_copies":14240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8940,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1274}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8934","reported_seq":8992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.727315+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:26.727315+0000","last_peered":"2024-09-15T21:10:26.727315+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:26.727315+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652812,"num_objects":3498,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9024","reported_seq":9108,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:23.610859+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:23.610859+0000","last_peered":"2024-09-15T21:10:23.610859+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:23.610859+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29955340,"num_objects":3657,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1376}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9024","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:25.916120+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:25.916120+0000","last_peered":"2024-09-15T21:10:25.916120+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:25.916120+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873996,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":462288686,"num_objects":56437,"num_object_clones":0,"num_object_copies":225748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56437,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137975,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":927363072,"data_stored":927363072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143888,"ondisk_log_size":143888,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721032,"kb_used_data":225656,"kb_used_omap":4,"kb_used_meta":495355,"kb_avail":93650808,"statfs":{"total":96636764160,"available":95898427392,"internally_reserved":0,"allocated":231071744,"data_stored":230906755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":507244240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":706092,"kb_used_data":227856,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":93665748,"statfs":{"total":96636764160,"available":95913725952,"internally_reserved":0,"allocated":233324544,"data_stored":233159555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":489683397},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,4,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051662,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":574932,"kb_used_data":228164,"kb_used_omap":4,"kb_used_meta":346747,"kb_avail":93796908,"statfs":{"total":96636764160,"available":96048033792,"internally_reserved":0,"allocated":233639936,"data_stored":233474947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":355069647},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":226540,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":231976960,"data_stored":231811971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231002112,"data_stored":231002112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233132032,"data_stored":233132032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":232980480,"data_stored":232980480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230248448,"data_stored":230248448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:29.641 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:29.641 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8865", 'reported_seq': 8912, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.861866+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:26.861866+0000', 'last_peered': '2024-09-15T21:10:26.861866+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:26.861866+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8865, 'log_dups_size': 0, 'ondisk_log_size': 8865, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28565504, 'num_objects': 3487, 'num_object_clones': 0, 'num_object_copies': 13948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3487, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8865, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1211}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'9097", 'reported_seq': 13470, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.162316+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:26.162316+0000', 'last_peered': '2024-09-15T21:10:26.162316+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:26.162316+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9097, 'log_dups_size': 0, 'ondisk_log_size': 9097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28300556, 'num_objects': 3455, 'num_object_clones': 0, 'num_object_copies': 13820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 229, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3455, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9095, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2155, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3226}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8825", 'reported_seq': 8906, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:23.775268+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:23.775268+0000', 'last_peered': '2024-09-15T21:10:23.775268+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:23.775268+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8825, 'log_dups_size': 0, 'ondisk_log_size': 8825, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29547166, 'num_objects': 3608, 'num_object_clones': 0, 'num_object_copies': 14432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3608, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8824, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1350}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'8985", 'reported_seq': 9055, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.057886+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:26.057886+0000', 'last_peered': '2024-09-15T21:10:26.057886+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:26.057886+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8985, 'log_dups_size': 0, 'ondisk_log_size': 8985, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28431628, 'num_objects': 3471, 'num_object_clones': 0, 'num_object_copies': 13884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3471, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8983, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1137}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:29.642 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'9112", 'reported_seq': 9169, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.898465+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:26.898465+0000', 'last_peered': '2024-09-15T21:10:26.898465+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:26.898465+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9112, 'log_dups_size': 0, 'ondisk_log_size': 9112, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29054220, 'num_objects': 3547, 'num_object_clones': 0, 'num_object_copies': 14188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9110, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1236}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8930", 'reported_seq': 8974, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.858065+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:26.858065+0000', 'last_peered': '2024-09-15T21:10:26.858065+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:26.858065+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8930, 'log_dups_size': 0, 'ondisk_log_size': 8930, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28229632, 'num_objects': 3446, 'num_object_clones': 0, 'num_object_copies': 13784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8930, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1130}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9142", 'reported_seq': 9212, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.084526+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:26.084526+0000', 'last_peered': '2024-09-15T21:10:26.084526+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:26.084526+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9142, 'log_dups_size': 0, 'ondisk_log_size': 9142, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29734156, 'num_objects': 3630, 'num_object_clones': 0, 'num_object_copies': 14520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9140, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1264}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9249", 'reported_seq': 9319, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.075143+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:26.075143+0000', 'last_peered': '2024-09-15T21:10:26.075143+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:26.075143+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9249, 'log_dups_size': 0, 'ondisk_log_size': 9249, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28767500, 'num_objects': 3512, 'num_object_clones': 0, 'num_object_copies': 14048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9247, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1038}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:29.643 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'8984", 'reported_seq': 9045, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.687071+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:26.687071+0000', 'last_peered': '2024-09-15T21:10:26.687071+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:26.687071+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8984, 'log_dups_size': 0, 'ondisk_log_size': 8984, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28404248, 'num_objects': 3468, 'num_object_clones': 0, 'num_object_copies': 13872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3468, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8981, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1136}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'9022", 'reported_seq': 9104, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:23.649892+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:23.649892+0000', 'last_peered': '2024-09-15T21:10:23.649892+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:23.649892+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9022, 'log_dups_size': 0, 'ondisk_log_size': 9022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28996876, 'num_objects': 3540, 'num_object_clones': 0, 'num_object_copies': 14160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9020, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1205}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8904", 'reported_seq': 8945, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.865076+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:26.865076+0000', 'last_peered': '2024-09-15T21:10:26.865076+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:26.865076+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8904, 'log_dups_size': 0, 'ondisk_log_size': 8904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28859014, 'num_objects': 3523, 'num_object_clones': 0, 'num_object_copies': 14092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8903, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1267}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'8941", 'reported_seq': 9022, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:23.630939+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:23.630939+0000', 'last_peered': '2024-09-15T21:10:23.630939+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:23.630939+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8941, 'log_dups_size': 0, 'ondisk_log_size': 8941, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29162118, 'num_objects': 3560, 'num_object_clones': 0, 'num_object_copies': 14240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3560, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8940, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1274}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'8934", 'reported_seq': 8992, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:26.727315+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:26.727315+0000', 'last_peered': '2024-09-15T21:10:26.727315+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:26.727315+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8934, 'log_dups_size': 0, 'ondisk_log_size': 8934, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28652812, 'num_objects': 3498, 'num_object_clones': 0, 'num_object_copies': 13992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8932, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1219}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:29.644 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:29.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'9024", 'reported_seq': 9108, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:23.610859+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:23.610859+0000', 'last_peered': '2024-09-15T21:10:23.610859+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:23.610859+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9024, 'log_dups_size': 0, 'ondisk_log_size': 9024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29955340, 'num_objects': 3657, 'num_object_clones': 0, 'num_object_copies': 14628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9022, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1376}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:29.645 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:29.645 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'9024", 'reported_seq': 9094, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:25.916120+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:25.916120+0000', 'last_peered': '2024-09-15T21:10:25.916120+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:25.916120+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9024, 'log_dups_size': 0, 'ondisk_log_size': 9024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28873996, 'num_objects': 3525, 'num_object_clones': 0, 'num_object_copies': 14100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9022, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1210}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:29.645 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:30.389 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:30.389 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32670/224672 objects degraded (14.541%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":496,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":56168,"data_bytes":460085038,"bytes_used":2796257280,"bytes_avail":383750799360,"bytes_total":386547056640,"degraded_objects":32670,"degraded_total":224672,"degraded_ratio":0.1454119783506623,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":137,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":200},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:30.416 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:30.416 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:31.017 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:31.018 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:31.039 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-15T21:10:30.080738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":460085038,"num_objects":56168,"num_object_clones":0,"num_object_copies":224672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138244,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144157,"ondisk_log_size":144157,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2730720,"kb_used_data":899736,"kb_used_omap":14,"kb_used_meta":1830897,"kb_avail":374756640,"statfs":{"total":386547056640,"available":383750799360,"internally_reserved":0,"allocated":921329664,"data_stored":920669708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1874839090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,10,5,10,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19685376,"num_objects":-2403,"num_object_clones":0,"num_object_copies":-9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-137,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2403,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008647"},"pg_stats":[{"pgid":"1.f","version":"33'8865","reported_seq":8912,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.861866+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:26.861866+0000","last_peered":"2024-09-15T21:10:26.861866+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:26.861866+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8865,"log_dups_size":0,"ondisk_log_size":8865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28565504,"num_objects":3487,"num_object_clones":0,"num_object_copies":13948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1211}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9097","reported_seq":13470,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.162316+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:26.162316+0000","last_peered":"2024-09-15T21:10:26.162316+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:26.162316+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9097,"log_dups_size":0,"ondisk_log_size":9097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28300556,"num_objects":3455,"num_object_clones":0,"num_object_copies":13820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9095,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2155,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3226}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8896","reported_seq":8976,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:28.629960+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:28.629960+0000","last_peered":"2024-09-15T21:10:28.629960+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:28.629960+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8896,"log_dups_size":0,"ondisk_log_size":8896,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28965534,"num_objects":3537,"num_object_clones":0,"num_object_copies":14148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8895,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1279}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'8985","reported_seq":9055,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.057886+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:26.057886+0000","last_peered":"2024-09-15T21:10:26.057886+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:26.057886+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8985,"log_dups_size":0,"ondisk_log_size":8985,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28431628,"num_objects":3471,"num_object_clones":0,"num_object_copies":13884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8983,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1137}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9112","reported_seq":9169,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.898465+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:26.898465+0000","last_peered":"2024-09-15T21:10:26.898465+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:26.898465+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9112,"log_dups_size":0,"ondisk_log_size":9112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29054220,"num_objects":3547,"num_object_clones":0,"num_object_copies":14188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8930","reported_seq":8974,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.858065+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:26.858065+0000","last_peered":"2024-09-15T21:10:26.858065+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:26.858065+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8930,"log_dups_size":0,"ondisk_log_size":8930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28229632,"num_objects":3446,"num_object_clones":0,"num_object_copies":13784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8930,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9142","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.084526+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:26.084526+0000","last_peered":"2024-09-15T21:10:26.084526+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:26.084526+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9142,"log_dups_size":0,"ondisk_log_size":9142,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29734156,"num_objects":3630,"num_object_clones":0,"num_object_copies":14520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9140,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1264}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9249","reported_seq":9319,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.075143+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:26.075143+0000","last_peered":"2024-09-15T21:10:26.075143+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:26.075143+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9249,"log_dups_size":0,"ondisk_log_size":9249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28767500,"num_objects":3512,"num_object_clones":0,"num_object_copies":14048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1038}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'8984","reported_seq":9045,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.687071+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:26.687071+0000","last_peered":"2024-09-15T21:10:26.687071+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:26.687071+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8984,"log_dups_size":0,"ondisk_log_size":8984,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28404248,"num_objects":3468,"num_object_clones":0,"num_object_copies":13872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8981,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9098","reported_seq":9179,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:28.526060+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:28.526060+0000","last_peered":"2024-09-15T21:10:28.526060+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:28.526060+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28374284,"num_objects":3464,"num_object_clones":0,"num_object_copies":13856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9096,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8850","reported_seq":13514,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:26.749104+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:26.749104+0000","last_peered":"2024-09-15T21:10:26.749104+0000","last_clean":"2024-09-15T21:10:26.749104+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:26.749104+0000","last_undegraded":"2024-09-15T21:10:26.749104+0000","last_fullsized":"2024-09-15T21:10:26.749104+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8850,"log_dups_size":0,"ondisk_log_size":8850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28753920,"num_objects":3510,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2961,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8904","reported_seq":8945,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.865076+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:26.865076+0000","last_peered":"2024-09-15T21:10:26.865076+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:26.865076+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8904,"log_dups_size":0,"ondisk_log_size":8904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28859014,"num_objects":3523,"num_object_clones":0,"num_object_copies":14092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8903,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1267}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'8994","reported_seq":9075,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:28.563771+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:28.563771+0000","last_peered":"2024-09-15T21:10:28.563771+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:28.563771+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8994,"log_dups_size":0,"ondisk_log_size":8994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28727942,"num_objects":3507,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8993,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1221}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'8934","reported_seq":8992,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:26.727315+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:26.727315+0000","last_peered":"2024-09-15T21:10:26.727315+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:26.727315+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8934,"log_dups_size":0,"ondisk_log_size":8934,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28652812,"num_objects":3498,"num_object_clones":0,"num_object_copies":13992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8932,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1219}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9093","reported_seq":9176,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:28.811131+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:28.811131+0000","last_peered":"2024-09-15T21:10:28.811131+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:28.811131+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9093,"log_dups_size":0,"ondisk_log_size":9093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29390092,"num_objects":3588,"num_object_clones":0,"num_object_copies":14352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1307}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9024","reported_seq":9094,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:25.916120+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:25.916120+0000","last_peered":"2024-09-15T21:10:25.916120+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:25.916120+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9024,"log_dups_size":0,"ondisk_log_size":9024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28873996,"num_objects":3525,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1210}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":460085038,"num_objects":56168,"num_object_clones":0,"num_object_copies":224672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138244,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2177,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":918556672,"data_stored":918556672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144157,"ondisk_log_size":144157,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721032,"kb_used_data":225656,"kb_used_omap":4,"kb_used_meta":495355,"kb_avail":93650808,"statfs":{"total":96636764160,"available":95898427392,"internally_reserved":0,"allocated":231071744,"data_stored":230906755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":507244240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,4,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":724032,"kb_used_data":223596,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93647808,"statfs":{"total":96636764160,"available":95895355392,"internally_reserved":0,"allocated":228962304,"data_stored":228797315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576280,"kb_used_data":223944,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93795560,"statfs":{"total":96636764160,"available":96046653440,"internally_reserved":0,"allocated":229318656,"data_stored":229153667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247900,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709376,"kb_used_data":226540,"kb_used_omap":4,"kb_used_meta":482811,"kb_avail":93662464,"statfs":{"total":96636764160,"available":95910363136,"internally_reserved":0,"allocated":231976960,"data_stored":231811971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":494399182},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231002112,"data_stored":231002112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228675584,"data_stored":228675584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":228630528,"data_stored":228630528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230248448,"data_stored":230248448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:34.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:34.711 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:34.712 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:34.739 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-15T21:10:34.082062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749848,"kb_used_data":886928,"kb_used_omap":14,"kb_used_meta":1862833,"kb_avail":374737512,"statfs":{"total":386547056640,"available":383731212288,"internally_reserved":0,"allocated":908214272,"data_stored":907554316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1907541554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,13,12,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19603456,"num_objects":-2393,"num_object_clones":0,"num_object_copies":-9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005125"},"pg_stats":[{"pgid":"1.f","version":"33'8923","reported_seq":8969,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:32.020769+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:32.020769+0000","last_peered":"2024-09-15T21:10:32.020769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:32.020769+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8923,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9143","reported_seq":13697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.144334+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:31.144334+0000","last_peered":"2024-09-15T21:10:31.144334+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:31.144334+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923724,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2244,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8976","reported_seq":9057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.697034+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:33.697034+0000","last_peered":"2024-09-15T21:10:33.697034+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:33.697034+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28310174,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8975,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'9043","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.020891+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:31.020891+0000","last_peered":"2024-09-15T21:10:31.020891+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:31.020891+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956492,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9174","reported_seq":9231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.948822+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:31.948822+0000","last_peered":"2024-09-15T21:10:31.948822+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:31.948822+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1174}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8995","reported_seq":9041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.721231+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:31.721231+0000","last_peered":"2024-09-15T21:10:31.721231+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:31.721231+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8995,"log_dups_size":0,"ondisk_log_size":8995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27697152,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8995,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9201","reported_seq":9272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.178027+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:31.178027+0000","last_peered":"2024-09-15T21:10:31.178027+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:31.178027+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9201,"log_dups_size":0,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29250828,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1205}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9314","reported_seq":9384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.172746+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:31.172746+0000","last_peered":"2024-09-15T21:10:31.172746+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:31.172746+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28235020,"num_objects":3447,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9052","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.955218+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:31.955218+0000","last_peered":"2024-09-15T21:10:31.955218+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:31.955218+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27847192,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9162","reported_seq":9243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.806710+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:33.806710+0000","last_peered":"2024-09-15T21:10:33.806710+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:33.806710+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8916","reported_seq":13579,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:31.988420+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:31.988420+0000","last_peered":"2024-09-15T21:10:31.988420+0000","last_clean":"2024-09-15T21:10:31.988420+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:31.988420+0000","last_undegraded":"2024-09-15T21:10:31.988420+0000","last_fullsized":"2024-09-15T21:10:31.988420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8916,"log_dups_size":0,"ondisk_log_size":8916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8975","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.826518+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:31.826518+0000","last_peered":"2024-09-15T21:10:31.826518+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:31.826518+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277382,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9074","reported_seq":9155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.824362+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:33.824362+0000","last_peered":"2024-09-15T21:10:33.824362+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:33.824362+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28072582,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9008","reported_seq":9066,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.829355+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:31.829355+0000","last_peered":"2024-09-15T21:10:31.829355+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:31.829355+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046604,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9006,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1145}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9160","reported_seq":9244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.786082+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:33.786082+0000","last_peered":"2024-09-15T21:10:33.786082+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:33.786082+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28841228,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9092","reported_seq":9162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.125440+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:31.125440+0000","last_peered":"2024-09-15T21:10:31.125440+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:31.125440+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905428992,"data_stored":905428992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722340,"kb_used_data":221396,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649500,"statfs":{"total":96636764160,"available":95897088000,"internally_reserved":0,"allocated":226709504,"data_stored":226544515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":219340,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":224604160,"data_stored":224439171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576280,"kb_used_data":223944,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93795560,"statfs":{"total":96636764160,"available":96046653440,"internally_reserved":0,"allocated":229318656,"data_stored":229153667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727288,"kb_used_data":222248,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93644552,"statfs":{"total":96636764160,"available":95892021248,"internally_reserved":0,"allocated":227581952,"data_stored":227416963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,8,4,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226643968,"data_stored":226643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228675584,"data_stored":228675584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224210944,"data_stored":224210944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225898496,"data_stored":225898496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:34.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:35.376 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:35.377 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:35.401 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-15T21:10:34.082062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749848,"kb_used_data":886928,"kb_used_omap":14,"kb_used_meta":1862833,"kb_avail":374737512,"statfs":{"total":386547056640,"available":383731212288,"internally_reserved":0,"allocated":908214272,"data_stored":907554316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1907541554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,13,12,8,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19603456,"num_objects":-2393,"num_object_clones":0,"num_object_copies":-9572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2393,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005125"},"pg_stats":[{"pgid":"1.f","version":"33'8923","reported_seq":8969,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:32.020769+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:32.020769+0000","last_peered":"2024-09-15T21:10:32.020769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:32.020769+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8923,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9143","reported_seq":13697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.144334+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:31.144334+0000","last_peered":"2024-09-15T21:10:31.144334+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:31.144334+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923724,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2244,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8976","reported_seq":9057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.697034+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:33.697034+0000","last_peered":"2024-09-15T21:10:33.697034+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:33.697034+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28310174,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8975,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'9043","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.020891+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:31.020891+0000","last_peered":"2024-09-15T21:10:31.020891+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:31.020891+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956492,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9174","reported_seq":9231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.948822+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:31.948822+0000","last_peered":"2024-09-15T21:10:31.948822+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:31.948822+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1174}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8995","reported_seq":9041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.721231+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:31.721231+0000","last_peered":"2024-09-15T21:10:31.721231+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:31.721231+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8995,"log_dups_size":0,"ondisk_log_size":8995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27697152,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8995,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9201","reported_seq":9272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.178027+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:31.178027+0000","last_peered":"2024-09-15T21:10:31.178027+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:31.178027+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9201,"log_dups_size":0,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29250828,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1205}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9314","reported_seq":9384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.172746+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:31.172746+0000","last_peered":"2024-09-15T21:10:31.172746+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:31.172746+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28235020,"num_objects":3447,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9052","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.955218+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:31.955218+0000","last_peered":"2024-09-15T21:10:31.955218+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:31.955218+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27847192,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9162","reported_seq":9243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.806710+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:33.806710+0000","last_peered":"2024-09-15T21:10:33.806710+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:33.806710+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8916","reported_seq":13579,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:31.988420+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:31.988420+0000","last_peered":"2024-09-15T21:10:31.988420+0000","last_clean":"2024-09-15T21:10:31.988420+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:31.988420+0000","last_undegraded":"2024-09-15T21:10:31.988420+0000","last_fullsized":"2024-09-15T21:10:31.988420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8916,"log_dups_size":0,"ondisk_log_size":8916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8975","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.826518+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:31.826518+0000","last_peered":"2024-09-15T21:10:31.826518+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:31.826518+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277382,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9074","reported_seq":9155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.824362+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:33.824362+0000","last_peered":"2024-09-15T21:10:33.824362+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:33.824362+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28072582,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9008","reported_seq":9066,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.829355+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:31.829355+0000","last_peered":"2024-09-15T21:10:31.829355+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:31.829355+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046604,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9006,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1145}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9160","reported_seq":9244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.786082+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:33.786082+0000","last_peered":"2024-09-15T21:10:33.786082+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:33.786082+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28841228,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9092","reported_seq":9162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.125440+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:31.125440+0000","last_peered":"2024-09-15T21:10:31.125440+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:31.125440+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":905428992,"data_stored":905428992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722340,"kb_used_data":221396,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649500,"statfs":{"total":96636764160,"available":95897088000,"internally_reserved":0,"allocated":226709504,"data_stored":226544515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":219340,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":224604160,"data_stored":224439171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051663,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576280,"kb_used_data":223944,"kb_used_omap":4,"kb_used_meta":352315,"kb_avail":93795560,"statfs":{"total":96636764160,"available":96046653440,"internally_reserved":0,"allocated":229318656,"data_stored":229153667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":360771279},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,2,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727288,"kb_used_data":222248,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93644552,"statfs":{"total":96636764160,"available":95892021248,"internally_reserved":0,"allocated":227581952,"data_stored":227416963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,8,4,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226643968,"data_stored":226643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":228675584,"data_stored":228675584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224210944,"data_stored":224210944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225898496,"data_stored":225898496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:35.402 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8923", 'reported_seq': 8969, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:32.020769+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:32.020769+0000', 'last_peered': '2024-09-15T21:10:32.020769+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:32.020769+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8923, 'log_dups_size': 0, 'ondisk_log_size': 8923, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28090368, 'num_objects': 3429, 'num_object_clones': 0, 'num_object_copies': 13716, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3429, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8923, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1153}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'9143", 'reported_seq': 13697, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.144334+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:31.144334+0000', 'last_peered': '2024-09-15T21:10:31.144334+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:31.144334+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9143, 'log_dups_size': 0, 'ondisk_log_size': 9143, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27923724, 'num_objects': 3409, 'num_object_clones': 0, 'num_object_copies': 13636, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 140, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3409, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9141, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2244, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'8976", 'reported_seq': 9057, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:33.697034+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:33.697034+0000', 'last_peered': '2024-09-15T21:10:33.697034+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:33.697034+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8976, 'log_dups_size': 0, 'ondisk_log_size': 8976, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28310174, 'num_objects': 3457, 'num_object_clones': 0, 'num_object_copies': 13828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3457, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 8975, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1199}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'9043", 'reported_seq': 9113, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.020891+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:31.020891+0000', 'last_peered': '2024-09-15T21:10:31.020891+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:31.020891+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9043, 'log_dups_size': 0, 'ondisk_log_size': 9043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27956492, 'num_objects': 3413, 'num_object_clones': 0, 'num_object_copies': 13652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9041, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'9174", 'reported_seq': 9231, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.948822+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:31.948822+0000', 'last_peered': '2024-09-15T21:10:31.948822+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:31.948822+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9174, 'log_dups_size': 0, 'ondisk_log_size': 9174, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28546316, 'num_objects': 3485, 'num_object_clones': 0, 'num_object_copies': 13940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3485, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9172, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1174}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.403 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'8995", 'reported_seq': 9041, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.721231+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:31.721231+0000', 'last_peered': '2024-09-15T21:10:31.721231+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:31.721231+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8995, 'log_dups_size': 0, 'ondisk_log_size': 8995, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27697152, 'num_objects': 3381, 'num_object_clones': 0, 'num_object_copies': 13524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3381, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8995, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1065}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9201", 'reported_seq': 9272, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.178027+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:31.178027+0000', 'last_peered': '2024-09-15T21:10:31.178027+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:31.178027+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9201, 'log_dups_size': 0, 'ondisk_log_size': 9201, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 29250828, 'num_objects': 3571, 'num_object_clones': 0, 'num_object_copies': 14284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3571, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9199, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1205}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9314", 'reported_seq': 9384, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.172746+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:31.172746+0000', 'last_peered': '2024-09-15T21:10:31.172746+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:31.172746+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9314, 'log_dups_size': 0, 'ondisk_log_size': 9314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28235020, 'num_objects': 3447, 'num_object_clones': 0, 'num_object_copies': 13788, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3447, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9312, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 973}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'9052", 'reported_seq': 9113, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.955218+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:31.955218+0000', 'last_peered': '2024-09-15T21:10:31.955218+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:31.955218+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9052, 'log_dups_size': 0, 'ondisk_log_size': 9052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27847192, 'num_objects': 3400, 'num_object_clones': 0, 'num_object_copies': 13600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9049, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 1068}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'9162", 'reported_seq': 9243, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:33.806710+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:33.806710+0000', 'last_peered': '2024-09-15T21:10:33.806710+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:33.806710+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9162, 'log_dups_size': 0, 'ondisk_log_size': 9162, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27849996, 'num_objects': 3400, 'num_object_clones': 0, 'num_object_copies': 13600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3400, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9160, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1065}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:35.404 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'8975", 'reported_seq': 9016, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.826518+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:31.826518+0000', 'last_peered': '2024-09-15T21:10:31.826518+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:31.826518+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8975, 'log_dups_size': 0, 'ondisk_log_size': 8975, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28277382, 'num_objects': 3452, 'num_object_clones': 0, 'num_object_copies': 13808, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3452, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8974, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1196}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'9074", 'reported_seq': 9155, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:33.824362+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:33.824362+0000', 'last_peered': '2024-09-15T21:10:33.824362+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:33.824362+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9074, 'log_dups_size': 0, 'ondisk_log_size': 9074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28072582, 'num_objects': 3427, 'num_object_clones': 0, 'num_object_copies': 13708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3427, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9073, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1141}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'9008", 'reported_seq': 9066, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.829355+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:31.829355+0000', 'last_peered': '2024-09-15T21:10:31.829355+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:31.829355+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9008, 'log_dups_size': 0, 'ondisk_log_size': 9008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28046604, 'num_objects': 3424, 'num_object_clones': 0, 'num_object_copies': 13696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9006, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1145}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'9160", 'reported_seq': 9244, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:33.786082+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:33.786082+0000', 'last_peered': '2024-09-15T21:10:33.786082+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:33.786082+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9160, 'log_dups_size': 0, 'ondisk_log_size': 9160, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28841228, 'num_objects': 3521, 'num_object_clones': 0, 'num_object_copies': 14084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3521, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9158, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1240}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:35.405 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'9092", 'reported_seq': 9162, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:31.125440+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:31.125440+0000', 'last_peered': '2024-09-15T21:10:31.125440+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:31.125440+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9092, 'log_dups_size': 0, 'ondisk_log_size': 9092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28316940, 'num_objects': 3457, 'num_object_clones': 0, 'num_object_copies': 13828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3457, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9090, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1142}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:35.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:36.112 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:36.112 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32581/220468 objects degraded (14.778%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":501,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":15},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":55117,"data_bytes":451475246,"bytes_used":2815844352,"bytes_avail":383731212288,"bytes_total":386547056640,"degraded_objects":32581,"degraded_total":220468,"degraded_ratio":0.14778108387611807,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":199},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:36.140 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:36.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:36.770 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:36.771 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:36.798 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-15T21:10:36.082608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749612,"kb_used_data":882532,"kb_used_omap":14,"kb_used_meta":1866993,"kb_avail":374737748,"statfs":{"total":386547056640,"available":383731453952,"internally_reserved":0,"allocated":903712768,"data_stored":903052812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1911801394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,7,13,10,3,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17252352,"num_objects":-2106,"num_object_clones":0,"num_object_copies":-8424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004813"},"pg_stats":[{"pgid":"1.f","version":"33'8923","reported_seq":8969,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:32.020769+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:32.020769+0000","last_peered":"2024-09-15T21:10:32.020769+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:32.020769+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8923,"log_dups_size":0,"ondisk_log_size":8923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28090368,"num_objects":3429,"num_object_clones":0,"num_object_copies":13716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8923,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9143","reported_seq":13697,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.144334+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:31.144334+0000","last_peered":"2024-09-15T21:10:31.144334+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:31.144334+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9143,"log_dups_size":0,"ondisk_log_size":9143,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27923724,"num_objects":3409,"num_object_clones":0,"num_object_copies":13636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9141,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2244,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'8976","reported_seq":9057,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.697034+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:33.697034+0000","last_peered":"2024-09-15T21:10:33.697034+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:33.697034+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8976,"log_dups_size":0,"ondisk_log_size":8976,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28310174,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8975,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1199}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'9043","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.020891+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:31.020891+0000","last_peered":"2024-09-15T21:10:31.020891+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:31.020891+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27956492,"num_objects":3413,"num_object_clones":0,"num_object_copies":13652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9041,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9174","reported_seq":9231,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.948822+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:31.948822+0000","last_peered":"2024-09-15T21:10:31.948822+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:31.948822+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9174,"log_dups_size":0,"ondisk_log_size":9174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28546316,"num_objects":3485,"num_object_clones":0,"num_object_copies":13940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1174}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'8995","reported_seq":9041,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.721231+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:31.721231+0000","last_peered":"2024-09-15T21:10:31.721231+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:31.721231+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8995,"log_dups_size":0,"ondisk_log_size":8995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27697152,"num_objects":3381,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8995,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1065}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9201","reported_seq":9272,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.178027+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:31.178027+0000","last_peered":"2024-09-15T21:10:31.178027+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:31.178027+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9201,"log_dups_size":0,"ondisk_log_size":9201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29250828,"num_objects":3571,"num_object_clones":0,"num_object_copies":14284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9199,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1205}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9314","reported_seq":9384,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.172746+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:31.172746+0000","last_peered":"2024-09-15T21:10:31.172746+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:31.172746+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28235020,"num_objects":3447,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":973}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9052","reported_seq":9113,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.955218+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:31.955218+0000","last_peered":"2024-09-15T21:10:31.955218+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:31.955218+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9052,"log_dups_size":0,"ondisk_log_size":9052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27847192,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9049,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":1068}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9162","reported_seq":9243,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.806710+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:33.806710+0000","last_peered":"2024-09-15T21:10:33.806710+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:33.806710+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9162,"log_dups_size":0,"ondisk_log_size":9162,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27849996,"num_objects":3400,"num_object_clones":0,"num_object_copies":13600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9160,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1065}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8916","reported_seq":13579,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:31.988420+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:31.988420+0000","last_peered":"2024-09-15T21:10:31.988420+0000","last_clean":"2024-09-15T21:10:31.988420+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:31.988420+0000","last_undegraded":"2024-09-15T21:10:31.988420+0000","last_fullsized":"2024-09-15T21:10:31.988420+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8916,"log_dups_size":0,"ondisk_log_size":8916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28213248,"num_objects":3444,"num_object_clones":0,"num_object_copies":13776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'8975","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.826518+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:31.826518+0000","last_peered":"2024-09-15T21:10:31.826518+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:31.826518+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8975,"log_dups_size":0,"ondisk_log_size":8975,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28277382,"num_objects":3452,"num_object_clones":0,"num_object_copies":13808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8974,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1196}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9074","reported_seq":9155,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.824362+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:33.824362+0000","last_peered":"2024-09-15T21:10:33.824362+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:33.824362+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28072582,"num_objects":3427,"num_object_clones":0,"num_object_copies":13708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1141}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9008","reported_seq":9066,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.829355+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:31.829355+0000","last_peered":"2024-09-15T21:10:31.829355+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:31.829355+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9008,"log_dups_size":0,"ondisk_log_size":9008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28046604,"num_objects":3424,"num_object_clones":0,"num_object_copies":13696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9006,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1145}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9160","reported_seq":9244,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:33.786082+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:33.786082+0000","last_peered":"2024-09-15T21:10:33.786082+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:33.786082+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9160,"log_dups_size":0,"ondisk_log_size":9160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28841228,"num_objects":3521,"num_object_clones":0,"num_object_copies":14084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9158,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1240}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9092","reported_seq":9162,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:31.125440+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:31.125440+0000","last_peered":"2024-09-15T21:10:31.125440+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:31.125440+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28316940,"num_objects":3457,"num_object_clones":0,"num_object_copies":13828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9090,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1142}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":451475246,"num_objects":55117,"num_object_clones":0,"num_object_copies":220468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32581,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139295,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2266,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":900915200,"data_stored":900915200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145208,"ondisk_log_size":145208,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722340,"kb_used_data":221396,"kb_used_omap":4,"kb_used_meta":500923,"kb_avail":93649500,"statfs":{"total":96636764160,"available":95897088000,"internally_reserved":0,"allocated":226709504,"data_stored":226544515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":512945872},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,2,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723940,"kb_used_data":219340,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93647900,"statfs":{"total":96636764160,"available":95895449600,"internally_reserved":0,"allocated":224604160,"data_stored":224439171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576044,"kb_used_data":219548,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93795796,"statfs":{"total":96636764160,"available":96046895104,"internally_reserved":0,"allocated":224817152,"data_stored":224652163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727288,"kb_used_data":222248,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93644552,"statfs":{"total":96636764160,"available":95892021248,"internally_reserved":0,"allocated":227581952,"data_stored":227416963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,8,4,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":226643968,"data_stored":226643968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224161792,"data_stored":224161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":224210944,"data_stored":224210944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":225898496,"data_stored":225898496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:39.801 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:40.412 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:40.412 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:40.440 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-15T21:10:40.083651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442857262,"num_objects":54065,"num_object_clones":0,"num_object_copies":216260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140347,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146260,"ondisk_log_size":146260,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749280,"kb_used_data":869720,"kb_used_omap":14,"kb_used_meta":1879473,"kb_avail":374738080,"statfs":{"total":386547056640,"available":383731793920,"internally_reserved":0,"allocated":890593280,"data_stored":889933324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1924580914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,6,6,13,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19431424,"num_objects":-2372,"num_object_clones":0,"num_object_copies":-9488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003560"},"pg_stats":[{"pgid":"1.f","version":"33'8969","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:37.040195+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:37.040195+0000","last_peered":"2024-09-15T21:10:37.040195+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:37.040195+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27713536,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9222","reported_seq":13920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.274955+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:36.274955+0000","last_peered":"2024-09-15T21:10:36.274955+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:36.274955+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276556,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'9046","reported_seq":9127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.743334+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:38.743334+0000","last_peered":"2024-09-15T21:10:38.743334+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:38.743334+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736734,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9045,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'9117","reported_seq":9187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.103898+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:36.103898+0000","last_peered":"2024-09-15T21:10:36.103898+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:36.103898+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9117,"log_dups_size":0,"ondisk_log_size":9117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350284,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9234","reported_seq":9291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.545427+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:36.545427+0000","last_peered":"2024-09-15T21:10:36.545427+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:36.545427+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054796,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9232,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9061","reported_seq":9107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.876135+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:36.876135+0000","last_peered":"2024-09-15T21:10:36.876135+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:36.876135+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9061,"log_dups_size":0,"ondisk_log_size":9061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9273","reported_seq":9343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.267369+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:36.267369+0000","last_peered":"2024-09-15T21:10:36.267369+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:36.267369+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28661004,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1133}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9371","reported_seq":9441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.106720+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:36.106720+0000","last_peered":"2024-09-15T21:10:36.106720+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:36.106720+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9371,"log_dups_size":0,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27768076,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9121","reported_seq":9182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.939298+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:36.939298+0000","last_peered":"2024-09-15T21:10:36.939298+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:36.939298+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27281944,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9226","reported_seq":9308,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.874675+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:38.874675+0000","last_peered":"2024-09-15T21:10:38.874675+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:38.874675+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27325708,"num_objects":3336,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1001}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8980","reported_seq":13644,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:36.854053+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:36.854053+0000","last_peered":"2024-09-15T21:10:36.854053+0000","last_clean":"2024-09-15T21:10:36.854053+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:36.854053+0000","last_undegraded":"2024-09-15T21:10:36.854053+0000","last_fullsized":"2024-09-15T21:10:36.854053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27688960,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'9041","reported_seq":9082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.899387+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:36.899387+0000","last_peered":"2024-09-15T21:10:36.899387+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:36.899387+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736710,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9131","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.801087+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:38.801087+0000","last_peered":"2024-09-15T21:10:38.801087+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:38.801087+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27605638,"num_objects":3370,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9074","reported_seq":9131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.998536+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:36.998536+0000","last_peered":"2024-09-15T21:10:36.998536+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:36.998536+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505932,"num_objects":3358,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9072,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9236","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.901408+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:38.901408+0000","last_peered":"2024-09-15T21:10:38.901408+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:38.901408+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218636,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9158","reported_seq":9228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.243944+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:36.243944+0000","last_peered":"2024-09-15T21:10:36.243944+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:36.243944+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776268,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442857262,"num_objects":54065,"num_object_clones":0,"num_object_copies":216260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140347,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887762944,"data_stored":887762944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146260,"ondisk_log_size":146260,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722252,"kb_used_data":217148,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93649588,"statfs":{"total":96636764160,"available":95897178112,"internally_reserved":0,"allocated":222359552,"data_stored":222194563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723900,"kb_used_data":215148,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93647940,"statfs":{"total":96636764160,"available":95895490560,"internally_reserved":0,"allocated":220311552,"data_stored":220146563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576044,"kb_used_data":219548,"kb_used_omap":4,"kb_used_meta":356475,"kb_avail":93795796,"statfs":{"total":96636764160,"available":96046895104,"internally_reserved":0,"allocated":224817152,"data_stored":224652163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":365031119},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727084,"kb_used_data":217876,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93644756,"statfs":{"total":96636764160,"available":95892230144,"internally_reserved":0,"allocated":223105024,"data_stored":222940035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222187520,"data_stored":222187520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":224161792,"data_stored":224161792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219873280,"data_stored":219873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221540352,"data_stored":221540352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:40.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:41.044 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:41.045 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:41.085 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-15T21:10:40.440800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":442857262,"num_objects":54065,"num_object_clones":0,"num_object_copies":216260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140347,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146260,"ondisk_log_size":146260,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2749268,"kb_used_data":865548,"kb_used_omap":14,"kb_used_meta":1883633,"kb_avail":374738092,"statfs":{"total":386547056640,"available":383731806208,"internally_reserved":0,"allocated":886321152,"data_stored":885661196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1928840754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,4,10,19,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17227776,"num_objects":-2103,"num_object_clones":0,"num_object_copies":-8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.360062"},"pg_stats":[{"pgid":"1.f","version":"33'8969","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:37.040195+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:37.040195+0000","last_peered":"2024-09-15T21:10:37.040195+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:37.040195+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27713536,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9222","reported_seq":13920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.274955+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:36.274955+0000","last_peered":"2024-09-15T21:10:36.274955+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:36.274955+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276556,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'9046","reported_seq":9127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.743334+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:38.743334+0000","last_peered":"2024-09-15T21:10:38.743334+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:38.743334+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736734,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9045,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"33'9117","reported_seq":9187,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.103898+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:36.103898+0000","last_peered":"2024-09-15T21:10:36.103898+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:36.103898+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9117,"log_dups_size":0,"ondisk_log_size":9117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27350284,"num_objects":3339,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9115,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1005}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9234","reported_seq":9291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.545427+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:36.545427+0000","last_peered":"2024-09-15T21:10:36.545427+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:36.545427+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054796,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9232,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9061","reported_seq":9107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.876135+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:36.876135+0000","last_peered":"2024-09-15T21:10:36.876135+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:36.876135+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9061,"log_dups_size":0,"ondisk_log_size":9061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"33'9273","reported_seq":9343,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.267369+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:36.267369+0000","last_peered":"2024-09-15T21:10:36.267369+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:36.267369+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28661004,"num_objects":3499,"num_object_clones":0,"num_object_copies":13996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1133}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"33'9371","reported_seq":9441,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.106720+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:36.106720+0000","last_peered":"2024-09-15T21:10:36.106720+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:36.106720+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9371,"log_dups_size":0,"ondisk_log_size":9371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27768076,"num_objects":3390,"num_object_clones":0,"num_object_copies":13560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9369,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":916}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9121","reported_seq":9182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.939298+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:36.939298+0000","last_peered":"2024-09-15T21:10:36.939298+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:36.939298+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27281944,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9226","reported_seq":9308,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.874675+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:38.874675+0000","last_peered":"2024-09-15T21:10:38.874675+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:38.874675+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27325708,"num_objects":3336,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1001}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8980","reported_seq":13644,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:36.854053+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:36.854053+0000","last_peered":"2024-09-15T21:10:36.854053+0000","last_clean":"2024-09-15T21:10:36.854053+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:36.854053+0000","last_undegraded":"2024-09-15T21:10:36.854053+0000","last_fullsized":"2024-09-15T21:10:36.854053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27688960,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'9041","reported_seq":9082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.899387+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:36.899387+0000","last_peered":"2024-09-15T21:10:36.899387+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:36.899387+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736710,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9131","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.801087+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:38.801087+0000","last_peered":"2024-09-15T21:10:38.801087+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:38.801087+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27605638,"num_objects":3370,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9074","reported_seq":9131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.998536+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:36.998536+0000","last_peered":"2024-09-15T21:10:36.998536+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:36.998536+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505932,"num_objects":3358,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9072,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9236","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.901408+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:38.901408+0000","last_peered":"2024-09-15T21:10:38.901408+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:38.901408+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218636,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"33'9158","reported_seq":9228,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.243944+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:36.243944+0000","last_peered":"2024-09-15T21:10:36.243944+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:36.243944+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9158,"log_dups_size":0,"ondisk_log_size":9158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27776268,"num_objects":3391,"num_object_clones":0,"num_object_copies":13564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9156,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1076}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":442857262,"num_objects":54065,"num_object_clones":0,"num_object_copies":216260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54065,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140347,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":883269632,"data_stored":883269632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146260,"ondisk_log_size":146260,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722252,"kb_used_data":217148,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93649588,"statfs":{"total":96636764160,"available":95897178112,"internally_reserved":0,"allocated":222359552,"data_stored":222194563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723900,"kb_used_data":215148,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93647940,"statfs":{"total":96636764160,"available":95895490560,"internally_reserved":0,"allocated":220311552,"data_stored":220146563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576032,"kb_used_data":215376,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93795808,"statfs":{"total":96636764160,"available":96046907392,"internally_reserved":0,"allocated":220545024,"data_stored":220380035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247902,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727084,"kb_used_data":217876,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93644756,"statfs":{"total":96636764160,"available":95892230144,"internally_reserved":0,"allocated":223105024,"data_stored":222940035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,4,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":222187520,"data_stored":222187520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219668480,"data_stored":219668480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219873280,"data_stored":219873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221540352,"data_stored":221540352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "33'8969", 'reported_seq': 9016, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:37.040195+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:37.040195+0000', 'last_peered': '2024-09-15T21:10:37.040195+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:37.040195+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 8969, 'log_dups_size': 0, 'ondisk_log_size': 8969, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27713536, 'num_objects': 3383, 'num_object_clones': 0, 'num_object_copies': 13532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3383, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 8969, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1107}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "33'9222", 'reported_seq': 13920, 'reported_epoch': 33, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.274955+0000', 'last_change': '2024-09-15T21:07:00.020178+0000', 'last_active': '2024-09-15T21:10:36.274955+0000', 'last_peered': '2024-09-15T21:10:36.274955+0000', 'last_clean': '2024-09-15T21:02:50.213249+0000', 'last_became_active': '2024-09-15T21:03:59.612089+0000', 'last_became_peered': '2024-09-15T21:03:59.612089+0000', 'last_unstale': '2024-09-15T21:10:36.274955+0000', 'last_undegraded': '2024-09-15T21:03:59.106699+0000', 'last_fullsized': '2024-09-15T21:03:59.045571+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9222, 'log_dups_size': 0, 'ondisk_log_size': 9222, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27276556, 'num_objects': 3330, 'num_object_clones': 0, 'num_object_copies': 13320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 68, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9220, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2316, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [2147483647, 0, 3, 2], 'avail_no_missing': ['0(1)', '2(3)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(3),3(2)', 'objects': 3262}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:41.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "33'9046", 'reported_seq': 9127, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:38.743334+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:38.743334+0000', 'last_peered': '2024-09-15T21:10:38.743334+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:38.743334+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9046, 'log_dups_size': 0, 'ondisk_log_size': 9046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27736734, 'num_objects': 3387, 'num_object_clones': 0, 'num_object_copies': 13548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3387, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9045, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1129}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "33'9117", 'reported_seq': 9187, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.103898+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:36.103898+0000', 'last_peered': '2024-09-15T21:10:36.103898+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:36.103898+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9117, 'log_dups_size': 0, 'ondisk_log_size': 9117, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27350284, 'num_objects': 3339, 'num_object_clones': 0, 'num_object_copies': 13356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3339, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9115, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "33'9234", 'reported_seq': 9291, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.545427+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:36.545427+0000', 'last_peered': '2024-09-15T21:10:36.545427+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:36.545427+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9234, 'log_dups_size': 0, 'ondisk_log_size': 9234, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28054796, 'num_objects': 3425, 'num_object_clones': 0, 'num_object_copies': 13700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9232, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1114}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "33'9061", 'reported_seq': 9107, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.876135+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:36.876135+0000', 'last_peered': '2024-09-15T21:10:36.876135+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:36.876135+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9061, 'log_dups_size': 0, 'ondisk_log_size': 9061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27156480, 'num_objects': 3315, 'num_object_clones': 0, 'num_object_copies': 13260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9061, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 999}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "33'9273", 'reported_seq': 9343, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.267369+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:36.267369+0000', 'last_peered': '2024-09-15T21:10:36.267369+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:36.267369+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9273, 'log_dups_size': 0, 'ondisk_log_size': 9273, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28661004, 'num_objects': 3499, 'num_object_clones': 0, 'num_object_copies': 13996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9271, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1133}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "33'9371", 'reported_seq': 9441, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.106720+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:36.106720+0000', 'last_peered': '2024-09-15T21:10:36.106720+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:36.106720+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9371, 'log_dups_size': 0, 'ondisk_log_size': 9371, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27768076, 'num_objects': 3390, 'num_object_clones': 0, 'num_object_copies': 13560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3390, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9369, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 916}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:41.087 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "33'9121", 'reported_seq': 9182, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.939298+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:36.939298+0000', 'last_peered': '2024-09-15T21:10:36.939298+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:36.939298+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9121, 'log_dups_size': 0, 'ondisk_log_size': 9121, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27281944, 'num_objects': 3331, 'num_object_clones': 0, 'num_object_copies': 13324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9118, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 999}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "33'9226", 'reported_seq': 9308, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:38.874675+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:38.874675+0000', 'last_peered': '2024-09-15T21:10:38.874675+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:38.874675+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9226, 'log_dups_size': 0, 'ondisk_log_size': 9226, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27325708, 'num_objects': 3336, 'num_object_clones': 0, 'num_object_copies': 13344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9224, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 1001}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "33'9041", 'reported_seq': 9082, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.899387+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:36.899387+0000', 'last_peered': '2024-09-15T21:10:36.899387+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:36.899387+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9041, 'log_dups_size': 0, 'ondisk_log_size': 9041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27736710, 'num_objects': 3386, 'num_object_clones': 0, 'num_object_copies': 13544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9040, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1130}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "33'9131", 'reported_seq': 9212, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:38.801087+0000', 'last_change': '2024-09-15T21:03:59.682630+0000', 'last_active': '2024-09-15T21:10:38.801087+0000', 'last_peered': '2024-09-15T21:10:38.801087+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:38.801087+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9131, 'log_dups_size': 0, 'ondisk_log_size': 9131, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27605638, 'num_objects': 3370, 'num_object_clones': 0, 'num_object_copies': 13480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2287, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3370, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9130, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1084}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "33'9074", 'reported_seq': 9131, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.998536+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:36.998536+0000', 'last_peered': '2024-09-15T21:10:36.998536+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:36.998536+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9074, 'log_dups_size': 0, 'ondisk_log_size': 9074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27505932, 'num_objects': 3358, 'num_object_clones': 0, 'num_object_copies': 13432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3358, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9072, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "33'9236", 'reported_seq': 9320, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:38.901408+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:38.901408+0000', 'last_peered': '2024-09-15T21:10:38.901408+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:38.901408+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9236, 'log_dups_size': 0, 'ondisk_log_size': 9236, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28218636, 'num_objects': 3445, 'num_object_clones': 0, 'num_object_copies': 13780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3445, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9234, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1164}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:41.088 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:41.089 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "33'9158", 'reported_seq': 9228, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:36.243944+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:36.243944+0000', 'last_peered': '2024-09-15T21:10:36.243944+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:36.243944+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9158, 'log_dups_size': 0, 'ondisk_log_size': 9158, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27776268, 'num_objects': 3391, 'num_object_clones': 0, 'num_object_copies': 13564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9156, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1076}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:41.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:41.780 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:41.780 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32509/216260 objects degraded (15.032%), 15 pgs degraded, 15 pgs undersized","count":30},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":507,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":14},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":54065,"data_bytes":442857262,"bytes_used":2815262720,"bytes_avail":383731793920,"bytes_total":386547056640,"degraded_objects":32509,"degraded_total":216260,"degraded_ratio":0.1503236844538981,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":161,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:41.804 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:41.796+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9294 (0'0,35'9294] local-lis/les=34/35 n=3258 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9284 crt=35'9284 lcod 35'9283 mlcod 35'9283 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:41.805 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:41.806 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:42.013 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.006+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9294 (0'0,35'9294] local-lis/les=34/35 n=3258 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9294 lcod 35'9293 mlcod 35'9293 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:42.072 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.065+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9294 (0'0,35'9294] local-lis/les=34/35 n=3258 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9294 lcod 35'9293 mlcod 35'9293 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:42.341 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.337+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9298 (0'0,35'9298] local-lis/les=34/35 n=3254 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9297 crt=35'9297 lcod 35'9296 mlcod 35'9296 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:42.400 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:42.400 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:42.424 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.419+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9298 (0'0,35'9298] local-lis/les=34/35 n=3254 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9298 lcod 35'9297 mlcod 35'9297 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:42.431 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2024-09-15T21:10:42.084320+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":440416046,"num_objects":53767,"num_object_clones":0,"num_object_copies":215068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140645,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146558,"ondisk_log_size":146558,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2732580,"kb_used_data":861340,"kb_used_omap":14,"kb_used_meta":1871153,"kb_avail":374754780,"statfs":{"total":386547056640,"available":383748894720,"internally_reserved":0,"allocated":882012160,"data_stored":881348158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1916061234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,8,9,13,18,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11059200,"num_objects":-1350,"num_object_clones":0,"num_object_copies":-5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1350,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002258"},"pg_stats":[{"pgid":"1.f","version":"33'8969","reported_seq":9016,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:37.040195+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:37.040195+0000","last_peered":"2024-09-15T21:10:37.040195+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:37.040195+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":8969,"log_dups_size":0,"ondisk_log_size":8969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27713536,"num_objects":3383,"num_object_clones":0,"num_object_copies":13532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8969,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1107}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9222","reported_seq":13920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.274955+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:36.274955+0000","last_peered":"2024-09-15T21:10:36.274955+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:36.274955+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276556,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"33'9046","reported_seq":9127,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.743334+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:38.743334+0000","last_peered":"2024-09-15T21:10:38.743334+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:38.743334+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9046,"log_dups_size":0,"ondisk_log_size":9046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736734,"num_objects":3387,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3387,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9045,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1129}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'9194","reported_seq":9266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.223570+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:41.223570+0000","last_peered":"2024-09-15T21:10:41.223570+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:41.223570+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9192,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"33'9234","reported_seq":9291,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.545427+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:36.545427+0000","last_peered":"2024-09-15T21:10:36.545427+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:36.545427+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28054796,"num_objects":3425,"num_object_clones":0,"num_object_copies":13700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9232,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1114}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"33'9061","reported_seq":9107,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.876135+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:36.876135+0000","last_peered":"2024-09-15T21:10:36.876135+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:36.876135+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9061,"log_dups_size":0,"ondisk_log_size":9061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27156480,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9061,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'9350","reported_seq":9423,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.177454+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:41.177454+0000","last_peered":"2024-09-15T21:10:41.177454+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:41.177454+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030220,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9348,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'9446","reported_seq":9518,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.153568+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:41.153568+0000","last_peered":"2024-09-15T21:10:41.153568+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:41.153568+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27153676,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"33'9121","reported_seq":9182,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.939298+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:36.939298+0000","last_peered":"2024-09-15T21:10:36.939298+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:36.939298+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9121,"log_dups_size":0,"ondisk_log_size":9121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27281944,"num_objects":3331,"num_object_clones":0,"num_object_copies":13324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9118,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":999}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"33'9226","reported_seq":9308,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.874675+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:38.874675+0000","last_peered":"2024-09-15T21:10:38.874675+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:38.874675+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9226,"log_dups_size":0,"ondisk_log_size":9226,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27325708,"num_objects":3336,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":1001}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"33'8980","reported_seq":13644,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-15T21:10:36.854053+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:36.854053+0000","last_peered":"2024-09-15T21:10:36.854053+0000","last_clean":"2024-09-15T21:10:36.854053+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:36.854053+0000","last_undegraded":"2024-09-15T21:10:36.854053+0000","last_fullsized":"2024-09-15T21:10:36.854053+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27688960,"num_objects":3380,"num_object_clones":0,"num_object_copies":13520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3091,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"33'9041","reported_seq":9082,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.899387+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:36.899387+0000","last_peered":"2024-09-15T21:10:36.899387+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:36.899387+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9041,"log_dups_size":0,"ondisk_log_size":9041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27736710,"num_objects":3386,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9040,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1130}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"33'9131","reported_seq":9212,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.801087+0000","last_change":"2024-09-15T21:03:59.682630+0000","last_active":"2024-09-15T21:10:38.801087+0000","last_peered":"2024-09-15T21:10:38.801087+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:38.801087+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9131,"log_dups_size":0,"ondisk_log_size":9131,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27605638,"num_objects":3370,"num_object_clones":0,"num_object_copies":13480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9130,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1084}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"33'9074","reported_seq":9131,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.998536+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:36.998536+0000","last_peered":"2024-09-15T21:10:36.998536+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:36.998536+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9074,"log_dups_size":0,"ondisk_log_size":9074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27505932,"num_objects":3358,"num_object_clones":0,"num_object_copies":13432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9072,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"33'9236","reported_seq":9320,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:38.901408+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:38.901408+0000","last_peered":"2024-09-15T21:10:38.901408+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:38.901408+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9236,"log_dups_size":0,"ondisk_log_size":9236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28218636,"num_objects":3445,"num_object_clones":0,"num_object_copies":13780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1164}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9227","reported_seq":9299,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.053436+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:41.053436+0000","last_peered":"2024-09-15T21:10:41.053436+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:41.053436+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":440416046,"num_objects":53767,"num_object_clones":0,"num_object_copies":215068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140645,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":878977024,"data_stored":878977024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146558,"ondisk_log_size":146558,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722252,"kb_used_data":217148,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93649588,"statfs":{"total":96636764160,"available":95897178112,"internally_reserved":0,"allocated":222359552,"data_stored":222194563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":723900,"kb_used_data":215148,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93647940,"statfs":{"total":96636764160,"available":95895490560,"internally_reserved":0,"allocated":220311552,"data_stored":220146563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576032,"kb_used_data":215376,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93795808,"statfs":{"total":96636764160,"available":96046907392,"internally_reserved":0,"allocated":220545024,"data_stored":220380035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710396,"kb_used_data":213668,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93661444,"statfs":{"total":96636764160,"available":95909318656,"internally_reserved":0,"allocated":218796032,"data_stored":218626997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217894912,"data_stored":217894912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219668480,"data_stored":219668480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":219873280,"data_stored":219873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221540352,"data_stored":221540352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:42.583 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.577+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9302 (0'0,35'9302] local-lis/les=34/35 n=3250 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9298 crt=35'9298 lcod 35'9297 mlcod 35'9297 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:42.804 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:42.796+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9305 (0'0,35'9305] local-lis/les=34/35 n=3247 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9304 crt=35'9303 lcod 35'9303 mlcod 35'9303 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.035 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.032+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9311 (0'0,35'9311] local-lis/les=34/35 n=3241 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9308 crt=35'9308 lcod 35'9307 mlcod 35'9307 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.148 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.142+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9312 (0'0,35'9312] local-lis/les=34/35 n=3240 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9312 lcod 35'9311 mlcod 35'9311 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.291 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.280+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9313 (0'0,35'9313] local-lis/les=34/35 n=3239 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9312 crt=35'9312 lcod 35'9311 mlcod 35'9311 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.434 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.425+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9314 (0'0,35'9314] local-lis/les=34/35 n=3238 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9313 crt=35'9313 lcod 35'9312 mlcod 35'9312 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.573 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.570+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9316 (0'0,35'9316] local-lis/les=34/35 n=3236 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9315 crt=35'9315 lcod 35'9314 mlcod 35'9314 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.765 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.761+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9319 (0'0,35'9319] local-lis/les=34/35 n=3233 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9317 crt=35'9317 lcod 35'9316 mlcod 35'9316 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.888 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.884+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9319 (0'0,35'9319] local-lis/les=34/35 n=3233 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9319 lcod 35'9318 mlcod 35'9318 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:43.992 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:43.985+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9320 (0'0,35'9320] local-lis/les=34/35 n=3232 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9319 crt=35'9319 lcod 35'9318 mlcod 35'9318 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.138 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.132+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9323 (0'0,35'9323] local-lis/les=34/35 n=3229 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9323 lcod 35'9322 mlcod 35'9322 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.305 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.299+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9325 (0'0,35'9325] local-lis/les=34/35 n=3227 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9324 crt=35'9324 lcod 35'9323 mlcod 35'9323 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.391 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.384+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9327 (0'0,35'9327] local-lis/les=34/35 n=3225 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9325 crt=35'9325 lcod 35'9324 mlcod 35'9324 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.573 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.562+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9330 (0'0,35'9330] local-lis/les=34/35 n=3222 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9327 crt=35'9327 lcod 35'9326 mlcod 35'9326 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.786 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.779+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9334 (0'0,35'9334] local-lis/les=34/35 n=3218 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9333 crt=35'9333 lcod 35'9332 mlcod 35'9332 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:44.984 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:44.981+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9338 (0'0,35'9338] local-lis/les=34/35 n=3214 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9335 crt=35'9334 lcod 35'9334 mlcod 35'9334 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:45.338 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:45.334+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9345 (0'0,35'9345] local-lis/les=34/35 n=3207 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9340 crt=35'9339 lcod 35'9339 mlcod 35'9339 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:45.431 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:45.426+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9346 (0'0,35'9346] local-lis/les=34/35 n=3206 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9345 crt=35'9342 lcod 35'9344 mlcod 35'9344 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:45.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:45.522 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:45.518+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9346 (0'0,35'9346] local-lis/les=34/35 n=3206 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9346 lcod 35'9345 mlcod 35'9345 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:45.642 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:45.638+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9347 (0'0,35'9347] local-lis/les=34/35 n=3205 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9347 lcod 35'9346 mlcod 35'9346 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:45.928 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:45.925+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9350 (0'0,35'9350] local-lis/les=34/35 n=3202 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9349 crt=35'9348 lcod 35'9348 mlcod 35'9348 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.058 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:46.059 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:46.069 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.063+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9351 (0'0,35'9351] local-lis/les=34/35 n=3201 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9351 lcod 35'9350 mlcod 35'9350 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.096 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-15T21:10:44.085077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":434878254,"num_objects":53091,"num_object_clones":0,"num_object_copies":212364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141321,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2388,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147234,"ondisk_log_size":147234,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716188,"kb_used_data":853244,"kb_used_omap":14,"kb_used_meta":1862833,"kb_avail":374771172,"statfs":{"total":386547056640,"available":383765680128,"internally_reserved":0,"allocated":873721856,"data_stored":873041628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1907541554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,12,14,26,16,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16596992,"num_objects":-2026,"num_object_clones":0,"num_object_copies":-8104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002469"},"pg_stats":[{"pgid":"1.f","version":"35'9033","reported_seq":9083,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.938164+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:41.938164+0000","last_peered":"2024-09-15T21:10:41.938164+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:41.938164+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"33'9222","reported_seq":13920,"reported_epoch":33,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:36.274955+0000","last_change":"2024-09-15T21:07:00.020178+0000","last_active":"2024-09-15T21:10:36.274955+0000","last_peered":"2024-09-15T21:10:36.274955+0000","last_clean":"2024-09-15T21:02:50.213249+0000","last_became_active":"2024-09-15T21:03:59.612089+0000","last_became_peered":"2024-09-15T21:03:59.612089+0000","last_unstale":"2024-09-15T21:10:36.274955+0000","last_undegraded":"2024-09-15T21:03:59.106699+0000","last_fullsized":"2024-09-15T21:03:59.045571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9222,"log_dups_size":0,"ondisk_log_size":9222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27276556,"num_objects":3330,"num_object_clones":0,"num_object_copies":13320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9220,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[2147483647,0,3,2],"avail_no_missing":["0(1)","2(3)","3(2)"],"object_location_counts":[{"shards":"0(1),2(3),3(2)","objects":3262}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"35'9106","reported_seq":9191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.816069+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:43.816069+0000","last_peered":"2024-09-15T21:10:43.816069+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:43.816069+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27245214,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9105,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'9194","reported_seq":9266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.223570+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:41.223570+0000","last_peered":"2024-09-15T21:10:41.223570+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:41.223570+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9192,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9294","reported_seq":9355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.508471+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:41.508471+0000","last_peered":"2024-09-15T21:10:41.508471+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:41.508471+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9294,"log_dups_size":0,"ondisk_log_size":9294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27563276,"num_objects":3365,"num_object_clones":0,"num_object_copies":13460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9125","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:42.019671+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:42.019671+0000","last_peered":"2024-09-15T21:10:42.019671+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:42.019671+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9125,"log_dups_size":0,"ondisk_log_size":9125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26632192,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'9350","reported_seq":9423,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.177454+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:41.177454+0000","last_peered":"2024-09-15T21:10:41.177454+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:41.177454+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030220,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9348,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'9446","reported_seq":9518,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.153568+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:41.153568+0000","last_peered":"2024-09-15T21:10:41.153568+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:41.153568+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27153676,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9171","reported_seq":9236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.983218+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:41.983218+0000","last_peered":"2024-09-15T21:10:41.983218+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:41.983218+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9171,"log_dups_size":0,"ondisk_log_size":9171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26872344,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9290","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.829124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:43.829124+0000","last_peered":"2024-09-15T21:10:43.829124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:43.829124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801420,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9034","reported_seq":13702,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:42.068515+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:42.068515+0000","last_peered":"2024-09-15T21:10:42.068515+0000","last_clean":"2024-09-15T21:10:42.068515+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:42.068515+0000","last_undegraded":"2024-09-15T21:10:42.068515+0000","last_fullsized":"2024-09-15T21:10:42.068515+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9092","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.534334+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:41.534334+0000","last_peered":"2024-09-15T21:10:41.534334+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:41.534334+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27318918,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9192","reported_seq":9378,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.944455+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:43.944455+0000","last_peered":"2024-09-15T21:10:43.944455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:43.944455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27105926,"num_objects":3309,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9156","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.974834+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:41.974834+0000","last_peered":"2024-09-15T21:10:41.974834+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:41.974834+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834188,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9302","reported_seq":9390,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.756554+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:43.756554+0000","last_peered":"2024-09-15T21:10:43.756554+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:43.756554+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9302,"log_dups_size":0,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677964,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9227","reported_seq":9299,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.053436+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:41.053436+0000","last_peered":"2024-09-15T21:10:41.053436+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:41.053436+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":434878254,"num_objects":53091,"num_object_clones":0,"num_object_copies":212364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32459,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53091,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141321,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2388,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":870748160,"data_stored":870748160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147234,"ondisk_log_size":147234,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722216,"kb_used_data":212944,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93649624,"statfs":{"total":96636764160,"available":95897214976,"internally_reserved":0,"allocated":218054656,"data_stored":217881554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,5,11,5,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707544,"kb_used_data":211256,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93664296,"statfs":{"total":96636764160,"available":95912239104,"internally_reserved":0,"allocated":216326144,"data_stored":216153042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051665,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576032,"kb_used_data":215376,"kb_used_omap":4,"kb_used_meta":360635,"kb_avail":93795808,"statfs":{"total":96636764160,"available":96046907392,"internally_reserved":0,"allocated":220545024,"data_stored":220380035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":369290959},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710396,"kb_used_data":213668,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93661444,"statfs":{"total":96636764160,"available":95909318656,"internally_reserved":0,"allocated":218796032,"data_stored":218626997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217894912,"data_stored":217894912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":219668480,"data_stored":219668480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215822336,"data_stored":215822336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217362432,"data_stored":217362432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:46.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:46.129 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.125+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9351 (0'0,35'9351] local-lis/les=34/35 n=3201 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9351 lcod 35'9350 mlcod 35'9350 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.212 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.194+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9351 (0'0,35'9351] local-lis/les=34/35 n=3201 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9351 lcod 35'9350 mlcod 35'9350 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.408 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.400+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9354 (0'0,35'9354] local-lis/les=34/35 n=3198 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9353 crt=35'9353 lcod 35'9352 mlcod 35'9352 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.682 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.678+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9355 (0'0,35'9355] local-lis/les=34/35 n=3197 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9355 lcod 35'9354 mlcod 35'9354 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:46.726 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:46.727 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:46.748 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-09-15T21:10:46.086022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":433870638,"num_objects":52968,"num_object_clones":0,"num_object_copies":211872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141444,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2456,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147357,"ondisk_log_size":147357,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2715944,"kb_used_data":848836,"kb_used_omap":14,"kb_used_meta":1866993,"kb_avail":374771416,"statfs":{"total":386547056640,"available":383765929984,"internally_reserved":0,"allocated":869208064,"data_stored":868519723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1911801394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,9,21,29,14,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11173888,"num_objects":-1364,"num_object_clones":0,"num_object_copies":-5456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002922"},"pg_stats":[{"pgid":"1.f","version":"35'9033","reported_seq":9083,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.938164+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:41.938164+0000","last_peered":"2024-09-15T21:10:41.938164+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:41.938164+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9033,"log_dups_size":0,"ondisk_log_size":9033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27189248,"num_objects":3319,"num_object_clones":0,"num_object_copies":13276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9033,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":1043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9345","reported_seq":14211,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:10:45.203730+0000","last_change":"2024-09-15T21:10:41.601447+0000","last_active":"2024-09-15T21:10:45.203730+0000","last_peered":"2024-09-15T21:10:45.203730+0000","last_clean":"2024-09-15T21:10:45.203730+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:45.203730+0000","last_undegraded":"2024-09-15T21:10:45.203730+0000","last_fullsized":"2024-09-15T21:10:45.203730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":1806,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268940,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2384,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9106","reported_seq":9191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.816069+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:43.816069+0000","last_peered":"2024-09-15T21:10:43.816069+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:43.816069+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27245214,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9105,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"34'9194","reported_seq":9266,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.223570+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:41.223570+0000","last_peered":"2024-09-15T21:10:41.223570+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:41.223570+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9194,"log_dups_size":0,"ondisk_log_size":9194,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26719500,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9192,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":928}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9294","reported_seq":9355,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.508471+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:41.508471+0000","last_peered":"2024-09-15T21:10:41.508471+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:41.508471+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9294,"log_dups_size":0,"ondisk_log_size":9294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27563276,"num_objects":3365,"num_object_clones":0,"num_object_copies":13460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":1054}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9125","reported_seq":9174,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:42.019671+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:42.019671+0000","last_peered":"2024-09-15T21:10:42.019671+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:42.019671+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9125,"log_dups_size":0,"ondisk_log_size":9125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26632192,"num_objects":3251,"num_object_clones":0,"num_object_copies":13004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":935}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"34'9350","reported_seq":9423,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.177454+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:41.177454+0000","last_peered":"2024-09-15T21:10:41.177454+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:41.177454+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9350,"log_dups_size":0,"ondisk_log_size":9350,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28030220,"num_objects":3422,"num_object_clones":0,"num_object_copies":13688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9348,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"34'9446","reported_seq":9518,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.153568+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:41.153568+0000","last_peered":"2024-09-15T21:10:41.153568+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:41.153568+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9446,"log_dups_size":0,"ondisk_log_size":9446,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27153676,"num_objects":3315,"num_object_clones":0,"num_object_copies":13260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":841}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9171","reported_seq":9236,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.983218+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:41.983218+0000","last_peered":"2024-09-15T21:10:41.983218+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:41.983218+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9171,"log_dups_size":0,"ondisk_log_size":9171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26872344,"num_objects":3281,"num_object_clones":0,"num_object_copies":13124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":949}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9290","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.829124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:43.829124+0000","last_peered":"2024-09-15T21:10:43.829124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:43.829124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801420,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9034","reported_seq":13702,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:42.068515+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:42.068515+0000","last_peered":"2024-09-15T21:10:42.068515+0000","last_clean":"2024-09-15T21:10:42.068515+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:42.068515+0000","last_undegraded":"2024-09-15T21:10:42.068515+0000","last_fullsized":"2024-09-15T21:10:42.068515+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9034,"log_dups_size":0,"ondisk_log_size":9034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27246592,"num_objects":3326,"num_object_clones":0,"num_object_copies":13304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3145,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9092","reported_seq":9137,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.534334+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:41.534334+0000","last_peered":"2024-09-15T21:10:41.534334+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:41.534334+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9092,"log_dups_size":0,"ondisk_log_size":9092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27318918,"num_objects":3335,"num_object_clones":0,"num_object_copies":13340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9091,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1079}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9192","reported_seq":9378,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.944455+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:43.944455+0000","last_peered":"2024-09-15T21:10:43.944455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:43.944455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27105926,"num_objects":3309,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9156","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.974834+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:41.974834+0000","last_peered":"2024-09-15T21:10:41.974834+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:41.974834+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9156,"log_dups_size":0,"ondisk_log_size":9156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26834188,"num_objects":3276,"num_object_clones":0,"num_object_copies":13104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9154,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":997}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9302","reported_seq":9390,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.756554+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:43.756554+0000","last_peered":"2024-09-15T21:10:43.756554+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:43.756554+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9302,"log_dups_size":0,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677964,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"34'9227","reported_seq":9299,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:41.053436+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:41.053436+0000","last_peered":"2024-09-15T21:10:41.053436+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:41.053436+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9227,"log_dups_size":0,"ondisk_log_size":9227,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9225,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":433870638,"num_objects":52968,"num_object_clones":0,"num_object_copies":211872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52968,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141444,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2456,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":866193408,"data_stored":866193408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147357,"ondisk_log_size":147357,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":722216,"kb_used_data":212944,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93649624,"statfs":{"total":96636764160,"available":95897214976,"internally_reserved":0,"allocated":218054656,"data_stored":217881554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,5,11,5,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707544,"kb_used_data":211256,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93664296,"statfs":{"total":96636764160,"available":95912239104,"internally_reserved":0,"allocated":216326144,"data_stored":216153042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575788,"kb_used_data":210968,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93796052,"statfs":{"total":96636764160,"available":96047157248,"internally_reserved":0,"allocated":216031232,"data_stored":215858130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247903,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710396,"kb_used_data":213668,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93661444,"statfs":{"total":96636764160,"available":95909318656,"internally_reserved":0,"allocated":218796032,"data_stored":218626997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,5,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217894912,"data_stored":217894912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215113728,"data_stored":215113728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215822336,"data_stored":215822336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217362432,"data_stored":217362432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:46.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:46.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9033", 'reported_seq': 9083, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.938164+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:41.938164+0000', 'last_peered': '2024-09-15T21:10:41.938164+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:41.938164+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9033, 'log_dups_size': 0, 'ondisk_log_size': 9033, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27189248, 'num_objects': 3319, 'num_object_clones': 0, 'num_object_copies': 13276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3319, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9033, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 1043}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.749 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.e is not active+clean 2024-09-15T21:10:46.749 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.e', 'version': "35'9345", 'reported_seq': 14211, 'reported_epoch': 35, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-15T21:10:45.203730+0000', 'last_change': '2024-09-15T21:10:41.601447+0000', 'last_active': '2024-09-15T21:10:45.203730+0000', 'last_peered': '2024-09-15T21:10:45.203730+0000', 'last_clean': '2024-09-15T21:10:45.203730+0000', 'last_became_active': '2024-09-15T21:10:41.499163+0000', 'last_became_peered': '2024-09-15T21:10:41.499163+0000', 'last_unstale': '2024-09-15T21:10:45.203730+0000', 'last_undegraded': '2024-09-15T21:10:45.203730+0000', 'last_fullsized': '2024-09-15T21:10:45.203730+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 35, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 1806, 'log_size': 9345, 'log_dups_size': 0, 'ondisk_log_size': 9345, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 3s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26268940, 'num_objects': 3207, 'num_object_clones': 0, 'num_object_copies': 12828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9343, 'num_write_kb': 50204, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2384, 'num_bytes_recovered': 13950976, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 3, 2], 'acting': [1, 0, 3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9106", 'reported_seq': 9191, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:43.816069+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:43.816069+0000', 'last_peered': '2024-09-15T21:10:43.816069+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:43.816069+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9106, 'log_dups_size': 0, 'ondisk_log_size': 9106, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27245214, 'num_objects': 3327, 'num_object_clones': 0, 'num_object_copies': 13308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3327, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9105, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1069}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "34'9194", 'reported_seq': 9266, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.223570+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:41.223570+0000', 'last_peered': '2024-09-15T21:10:41.223570+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:41.223570+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9194, 'log_dups_size': 0, 'ondisk_log_size': 9194, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26719500, 'num_objects': 3262, 'num_object_clones': 0, 'num_object_copies': 13048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3262, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9192, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 928}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9294", 'reported_seq': 9355, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.508471+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:41.508471+0000', 'last_peered': '2024-09-15T21:10:41.508471+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:41.508471+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9294, 'log_dups_size': 0, 'ondisk_log_size': 9294, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27563276, 'num_objects': 3365, 'num_object_clones': 0, 'num_object_copies': 13460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9292, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 1054}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9125", 'reported_seq': 9174, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:42.019671+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:42.019671+0000', 'last_peered': '2024-09-15T21:10:42.019671+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:42.019671+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9125, 'log_dups_size': 0, 'ondisk_log_size': 9125, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26632192, 'num_objects': 3251, 'num_object_clones': 0, 'num_object_copies': 13004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3251, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9125, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 935}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:46.750 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "34'9350", 'reported_seq': 9423, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.177454+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:41.177454+0000', 'last_peered': '2024-09-15T21:10:41.177454+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:41.177454+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9350, 'log_dups_size': 0, 'ondisk_log_size': 9350, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 28030220, 'num_objects': 3422, 'num_object_clones': 0, 'num_object_copies': 13688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3422, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9348, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 1056}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:46.751 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:46.751 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "34'9446", 'reported_seq': 9518, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.153568+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:41.153568+0000', 'last_peered': '2024-09-15T21:10:41.153568+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:41.153568+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9446, 'log_dups_size': 0, 'ondisk_log_size': 9446, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27153676, 'num_objects': 3315, 'num_object_clones': 0, 'num_object_copies': 13260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3315, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9444, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 841}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:46.751 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:46.751 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9171", 'reported_seq': 9236, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.983218+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:41.983218+0000', 'last_peered': '2024-09-15T21:10:41.983218+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:41.983218+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9171, 'log_dups_size': 0, 'ondisk_log_size': 9171, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26872344, 'num_objects': 3281, 'num_object_clones': 0, 'num_object_copies': 13124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3281, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9168, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 949}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9290", 'reported_seq': 9376, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:43.829124+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:43.829124+0000', 'last_peered': '2024-09-15T21:10:43.829124+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:43.829124+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9290, 'log_dups_size': 0, 'ondisk_log_size': 9290, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26801420, 'num_objects': 3272, 'num_object_clones': 0, 'num_object_copies': 13088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9288, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 937}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9092", 'reported_seq': 9137, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.534334+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:41.534334+0000', 'last_peered': '2024-09-15T21:10:41.534334+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:41.534334+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9092, 'log_dups_size': 0, 'ondisk_log_size': 9092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27318918, 'num_objects': 3335, 'num_object_clones': 0, 'num_object_copies': 13340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9091, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1079}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9192", 'reported_seq': 9378, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:43.944455+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:10:43.944455+0000', 'last_peered': '2024-09-15T21:10:43.944455+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:43.944455+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9192, 'log_dups_size': 0, 'ondisk_log_size': 9192, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27105926, 'num_objects': 3309, 'num_object_clones': 0, 'num_object_copies': 13236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2237, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3309, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9191, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1072}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9156", 'reported_seq': 9218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.974834+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:41.974834+0000', 'last_peered': '2024-09-15T21:10:41.974834+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:41.974834+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9156, 'log_dups_size': 0, 'ondisk_log_size': 9156, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26834188, 'num_objects': 3276, 'num_object_clones': 0, 'num_object_copies': 13104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3276, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9154, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 997}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:46.752 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:46.753 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9302", 'reported_seq': 9390, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:43.756554+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:43.756554+0000', 'last_peered': '2024-09-15T21:10:43.756554+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:43.756554+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9302, 'log_dups_size': 0, 'ondisk_log_size': 9302, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27677964, 'num_objects': 3379, 'num_object_clones': 0, 'num_object_copies': 13516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3379, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9300, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1098}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:46.753 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:46.753 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "34'9227", 'reported_seq': 9299, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:41.053436+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:41.053436+0000', 'last_peered': '2024-09-15T21:10:41.053436+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:41.053436+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9227, 'log_dups_size': 0, 'ondisk_log_size': 9227, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27211020, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 13288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9225, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 1007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:46.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:46.864 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:46.837+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9356 (0'0,35'9356] local-lis/les=34/35 n=3196 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9356 lcod 35'9355 mlcod 35'9355 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:47.143 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:47.136+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9361 (0'0,35'9361] local-lis/les=34/35 n=3191 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9359 crt=35'9359 lcod 35'9358 mlcod 35'9358 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:47.305 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:47.301+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9362 (0'0,35'9362] local-lis/les=34/35 n=3190 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9362 lcod 35'9361 mlcod 35'9361 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:47.394 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:47.388+0000 7f762e6f0640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9362 (0'0,35'9362] local-lis/les=34/35 n=3190 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9362 lcod 35'9361 mlcod 35'9361 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:47.540 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:47.541 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32391/211872 objects degraded (15.288%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+clean","count":1}],"num_pgs":16,"num_pools":1,"num_objects":52968,"data_bytes":433870638,"bytes_used":2781126656,"bytes_avail":383765929984,"bytes_total":386547056640,"degraded_objects":32391,"degraded_total":211872,"degraded_ratio":0.15288004077933848,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:47.562 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:10:47.562 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:47.587 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:47.583+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9366 (0'0,35'9366] local-lis/les=34/35 n=3186 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 luod=35'9363 crt=35'9363 lcod 35'9362 mlcod 35'9362 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:47.754 INFO:tasks.ceph.osd.1.smithi181.stderr:2024-09-15T21:10:47.749+0000 7f762a6e8640 -1 osd.1 pg_epoch: 35 pg[1.es0( v 35'9367 (0'0,35'9367] local-lis/les=34/35 n=3185 ec=13/13 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0,3,2]p1(0) r=0 lpr=34 crt=35'9367 lcod 35'9366 mlcod 35'9366 active+clean+scrubbing [ 1.es0: ] ] _scan_rollback_obs: queueing trans to clean up obsolete rollback objs 2024-09-15T21:10:48.197 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:48.198 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:48.226 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-15T21:10:48.089929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":428046126,"num_objects":52257,"num_object_clones":0,"num_object_copies":209028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142155,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2456,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148068,"ondisk_log_size":148068,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712352,"kb_used_data":841100,"kb_used_omap":14,"kb_used_meta":1871153,"kb_avail":374775008,"statfs":{"total":386547056640,"available":383769608192,"internally_reserved":0,"allocated":861286400,"data_stored":860593992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1916061234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,10,19,20,8,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14811136,"num_objects":-1808,"num_object_clones":0,"num_object_copies":-7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006278"},"pg_stats":[{"pgid":"1.f","version":"35'9106","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.966559+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:46.966559+0000","last_peered":"2024-09-15T21:10:46.966559+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:46.966559+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":3246,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9345","reported_seq":14211,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:10:45.203730+0000","last_change":"2024-09-15T21:10:41.601447+0000","last_active":"2024-09-15T21:10:45.203730+0000","last_peered":"2024-09-15T21:10:45.203730+0000","last_clean":"2024-09-15T21:10:45.203730+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:45.203730+0000","last_undegraded":"2024-09-15T21:10:45.203730+0000","last_fullsized":"2024-09-15T21:10:45.203730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":1806,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268940,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2384,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9106","reported_seq":9191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.816069+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:43.816069+0000","last_peered":"2024-09-15T21:10:43.816069+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:43.816069+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27245214,"num_objects":3327,"num_object_clones":0,"num_object_copies":13308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9105,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1069}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9258","reported_seq":9332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.236407+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:46.236407+0000","last_peered":"2024-09-15T21:10:46.236407+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:46.236407+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9258,"log_dups_size":0,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195212,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9353","reported_seq":9414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.943946+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:46.943946+0000","last_peered":"2024-09-15T21:10:46.943946+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:46.943946+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079948,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9182","reported_seq":9232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.112626+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:47.112626+0000","last_peered":"2024-09-15T21:10:47.112626+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:47.112626+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26165248,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9424","reported_seq":9498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.187085+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:46.187085+0000","last_peered":"2024-09-15T21:10:46.187085+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:46.187085+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9508","reported_seq":9582,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.288353+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:46.288353+0000","last_peered":"2024-09-15T21:10:46.288353+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:46.288353+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9508,"log_dups_size":0,"ondisk_log_size":9508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645772,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9240","reported_seq":9305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.031216+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:47.031216+0000","last_peered":"2024-09-15T21:10:47.031216+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:47.031216+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9240,"log_dups_size":0,"ondisk_log_size":9240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26307096,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9290","reported_seq":9376,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.829124+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:43.829124+0000","last_peered":"2024-09-15T21:10:43.829124+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:43.829124+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26801420,"num_objects":3272,"num_object_clones":0,"num_object_copies":13088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9288,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":937}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9098","reported_seq":13766,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:47.106908+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:47.106908+0000","last_peered":"2024-09-15T21:10:47.106908+0000","last_clean":"2024-09-15T21:10:47.106908+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:47.106908+0000","last_undegraded":"2024-09-15T21:10:47.106908+0000","last_fullsized":"2024-09-15T21:10:47.106908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3209,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9153","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.969254+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:46.969254+0000","last_peered":"2024-09-15T21:10:46.969254+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:46.969254+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819206,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9192","reported_seq":9378,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.944455+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:43.944455+0000","last_peered":"2024-09-15T21:10:43.944455+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:43.944455+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9192,"log_dups_size":0,"ondisk_log_size":9192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27105926,"num_objects":3309,"num_object_clones":0,"num_object_copies":13236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1072}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9216","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.993980+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:46.993980+0000","last_peered":"2024-09-15T21:10:46.993980+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:46.993980+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342668,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9302","reported_seq":9390,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:43.756554+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:43.756554+0000","last_peered":"2024-09-15T21:10:43.756554+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:43.756554+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9302,"log_dups_size":0,"ondisk_log_size":9302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27677964,"num_objects":3379,"num_object_clones":0,"num_object_copies":13516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9300,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1098}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9295","reported_seq":9369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.271180+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:46.271180+0000","last_peered":"2024-09-15T21:10:46.271180+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:46.271180+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26653964,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9293,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":428046126,"num_objects":52257,"num_object_clones":0,"num_object_copies":209028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32391,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142155,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2456,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":857956352,"data_stored":857956352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148068,"ondisk_log_size":148068,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718324,"kb_used_data":209060,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93653516,"statfs":{"total":96636764160,"available":95901200384,"internally_reserved":0,"allocated":214077440,"data_stored":213904338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,8,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707544,"kb_used_data":211256,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93664296,"statfs":{"total":96636764160,"available":95912239104,"internally_reserved":0,"allocated":216326144,"data_stored":216153042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,3,8,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575788,"kb_used_data":210968,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93796052,"statfs":{"total":96636764160,"available":96047157248,"internally_reserved":0,"allocated":216031232,"data_stored":215858130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710696,"kb_used_data":209816,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93661144,"statfs":{"total":96636764160,"available":95909011456,"internally_reserved":0,"allocated":214851584,"data_stored":214678482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,3,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213827584,"data_stored":213827584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215113728,"data_stored":215113728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":215822336,"data_stored":215822336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":213192704,"data_stored":213192704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:51.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:51.858 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:51.859 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:51.881 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-09-15T21:10:50.090471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":426235694,"num_objects":52036,"num_object_clones":0,"num_object_copies":208144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142376,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2511,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148289,"ondisk_log_size":148289,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712660,"kb_used_data":837260,"kb_used_omap":14,"kb_used_meta":1875313,"kb_avail":374774700,"statfs":{"total":386547056640,"available":383769292800,"internally_reserved":0,"allocated":857354240,"data_stored":856661832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1920321074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,5,16,19,10,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16621568,"num_objects":-2029,"num_object_clones":0,"num_object_copies":-8116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2029,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.649671"},"pg_stats":[{"pgid":"1.f","version":"35'9106","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.966559+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:46.966559+0000","last_peered":"2024-09-15T21:10:46.966559+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:46.966559+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":3246,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9345","reported_seq":14211,"reported_epoch":35,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:10:45.203730+0000","last_change":"2024-09-15T21:10:41.601447+0000","last_active":"2024-09-15T21:10:45.203730+0000","last_peered":"2024-09-15T21:10:45.203730+0000","last_clean":"2024-09-15T21:10:45.203730+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:45.203730+0000","last_undegraded":"2024-09-15T21:10:45.203730+0000","last_fullsized":"2024-09-15T21:10:45.203730+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":1806,"log_size":9345,"log_dups_size":0,"ondisk_log_size":9345,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26268940,"num_objects":3207,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9343,"num_write_kb":50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2384,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9153","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.564997+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:48.564997+0000","last_peered":"2024-09-15T21:10:48.564997+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:48.564997+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26860190,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9152,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9258","reported_seq":9332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.236407+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:46.236407+0000","last_peered":"2024-09-15T21:10:46.236407+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:46.236407+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9258,"log_dups_size":0,"ondisk_log_size":9258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26195212,"num_objects":3198,"num_object_clones":0,"num_object_copies":12792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9256,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":864}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9353","reported_seq":9414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.943946+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:46.943946+0000","last_peered":"2024-09-15T21:10:46.943946+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:46.943946+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079948,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9182","reported_seq":9232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.112626+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:47.112626+0000","last_peered":"2024-09-15T21:10:47.112626+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:47.112626+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26165248,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9424","reported_seq":9498,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.187085+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:46.187085+0000","last_peered":"2024-09-15T21:10:46.187085+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:46.187085+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9424,"log_dups_size":0,"ondisk_log_size":9424,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27424012,"num_objects":3348,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9422,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9508","reported_seq":9582,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.288353+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:46.288353+0000","last_peered":"2024-09-15T21:10:46.288353+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:46.288353+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9508,"log_dups_size":0,"ondisk_log_size":9508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26645772,"num_objects":3253,"num_object_clones":0,"num_object_copies":13012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":779}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9240","reported_seq":9305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.031216+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:47.031216+0000","last_peered":"2024-09-15T21:10:47.031216+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:47.031216+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9240,"log_dups_size":0,"ondisk_log_size":9240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26307096,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9347","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.906366+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:48.906366+0000","last_peered":"2024-09-15T21:10:48.906366+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:48.906366+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334476,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9098","reported_seq":13766,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:47.106908+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:47.106908+0000","last_peered":"2024-09-15T21:10:47.106908+0000","last_clean":"2024-09-15T21:10:47.106908+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:47.106908+0000","last_undegraded":"2024-09-15T21:10:47.106908+0000","last_fullsized":"2024-09-15T21:10:47.106908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3209,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9153","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.969254+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:46.969254+0000","last_peered":"2024-09-15T21:10:46.969254+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:46.969254+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819206,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9252","reported_seq":9548,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.888111+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:48.888111+0000","last_peered":"2024-09-15T21:10:48.888111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:48.888111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9252,"log_dups_size":0,"ondisk_log_size":9252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26614406,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9216","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.993980+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:46.993980+0000","last_peered":"2024-09-15T21:10:46.993980+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:46.993980+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342668,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9359","reported_seq":9447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.904424+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:48.904424+0000","last_peered":"2024-09-15T21:10:48.904424+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:48.904424+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9295","reported_seq":9369,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.271180+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:46.271180+0000","last_peered":"2024-09-15T21:10:46.271180+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:46.271180+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26653964,"num_objects":3254,"num_object_clones":0,"num_object_copies":13016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9293,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":939}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":426235694,"num_objects":52036,"num_object_clones":0,"num_object_copies":208144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52036,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142376,"num_write_kb":754111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2511,"num_bytes_recovered":13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":853839872,"data_stored":853839872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148289,"ondisk_log_size":148289,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718324,"kb_used_data":209060,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93653516,"statfs":{"total":96636764160,"available":95901200384,"internally_reserved":0,"allocated":214077440,"data_stored":213904338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,8,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707852,"kb_used_data":207416,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93663988,"statfs":{"total":96636764160,"available":95911923712,"internally_reserved":0,"allocated":212393984,"data_stored":212220882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051666,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575788,"kb_used_data":210968,"kb_used_omap":4,"kb_used_meta":364795,"kb_avail":93796052,"statfs":{"total":96636764160,"available":96047157248,"internally_reserved":0,"allocated":216031232,"data_stored":215858130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":373550799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,8,7,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247904,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710696,"kb_used_data":209816,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93661144,"statfs":{"total":96636764160,"available":95909011456,"internally_reserved":0,"allocated":214851584,"data_stored":214678482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,0,3,0,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":213827584,"data_stored":213827584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":215113728,"data_stored":215113728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211705856,"data_stored":211705856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":213192704,"data_stored":213192704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:51.882 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:52.515 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:52.516 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:52.539 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-15T21:10:52.091107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423698978,"num_objects":51726,"num_object_clones":0,"num_object_copies":206904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133352,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148597,"ondisk_log_size":148597,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713472,"kb_used_data":828340,"kb_used_omap":14,"kb_used_meta":1885041,"kb_avail":374773888,"statfs":{"total":386547056640,"available":383768461312,"internally_reserved":0,"allocated":848220160,"data_stored":847527752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1930282546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,9,3,11,12,7,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16717068,"num_objects":-2041,"num_object_clones":0,"num_object_copies":-8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7293,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2211,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.652829"},"pg_stats":[{"pgid":"1.f","version":"35'9106","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.966559+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:46.966559+0000","last_peered":"2024-09-15T21:10:46.966559+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:46.966559+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":3246,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9395","reported_seq":14288,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:50.072601+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:10:50.072601+0000","last_peered":"2024-09-15T21:10:50.072601+0000","last_clean":"2024-09-15T21:10:50.072601+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:50.072601+0000","last_undegraded":"2024-09-15T21:10:50.072601+0000","last_fullsized":"2024-09-15T21:10:50.072601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9395,"log_dups_size":0,"ondisk_log_size":9395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25845760,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9153","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.564997+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:48.564997+0000","last_peered":"2024-09-15T21:10:48.564997+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:48.564997+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26860190,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9152,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9321","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.305157+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:51.305157+0000","last_peered":"2024-09-15T21:10:51.305157+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:51.305157+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9353","reported_seq":9414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.943946+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:46.943946+0000","last_peered":"2024-09-15T21:10:46.943946+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:46.943946+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079948,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9182","reported_seq":9232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.112626+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:47.112626+0000","last_peered":"2024-09-15T21:10:47.112626+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:47.112626+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26165248,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9493","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.177145+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:51.177145+0000","last_peered":"2024-09-15T21:10:51.177145+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:51.177145+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858764,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9575","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.228316+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:51.228316+0000","last_peered":"2024-09-15T21:10:51.228316+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:51.228316+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096908,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9240","reported_seq":9305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.031216+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:47.031216+0000","last_peered":"2024-09-15T21:10:47.031216+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:47.031216+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9240,"log_dups_size":0,"ondisk_log_size":9240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26307096,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9347","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.906366+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:48.906366+0000","last_peered":"2024-09-15T21:10:48.906366+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:48.906366+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334476,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9098","reported_seq":13766,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:47.106908+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:47.106908+0000","last_peered":"2024-09-15T21:10:47.106908+0000","last_clean":"2024-09-15T21:10:47.106908+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:47.106908+0000","last_undegraded":"2024-09-15T21:10:47.106908+0000","last_fullsized":"2024-09-15T21:10:47.106908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3209,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9153","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.969254+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:46.969254+0000","last_peered":"2024-09-15T21:10:46.969254+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:46.969254+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819206,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9252","reported_seq":9548,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.888111+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:48.888111+0000","last_peered":"2024-09-15T21:10:48.888111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:48.888111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9252,"log_dups_size":0,"ondisk_log_size":9252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26614406,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9216","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.993980+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:46.993980+0000","last_peered":"2024-09-15T21:10:46.993980+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:46.993980+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342668,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9359","reported_seq":9447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.904424+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:48.904424+0000","last_peered":"2024-09-15T21:10:48.904424+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:48.904424+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9354","reported_seq":9428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.206952+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:51.206952+0000","last_peered":"2024-09-15T21:10:51.206952+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:51.206952+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26170636,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":423698978,"num_objects":51726,"num_object_clones":0,"num_object_copies":206904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133352,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":845393920,"data_stored":845393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148597,"ondisk_log_size":148597,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718324,"kb_used_data":209060,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93653516,"statfs":{"total":96636764160,"available":95901200384,"internally_reserved":0,"allocated":214077440,"data_stored":213904338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,8,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707852,"kb_used_data":207416,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93663988,"statfs":{"total":96636764160,"available":95911923712,"internally_reserved":0,"allocated":212393984,"data_stored":212220882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577008,"kb_used_data":206620,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93794832,"statfs":{"total":96636764160,"available":96045907968,"internally_reserved":0,"allocated":211578880,"data_stored":211405778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":379252431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710288,"kb_used_data":205244,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93661552,"statfs":{"total":96636764160,"available":95909429248,"internally_reserved":0,"allocated":210169856,"data_stored":209996754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,1,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209833984,"data_stored":209833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210661376,"data_stored":210661376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211705856,"data_stored":211705856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":213192704,"data_stored":213192704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9106", 'reported_seq': 9157, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:46.966559+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:46.966559+0000', 'last_peered': '2024-09-15T21:10:46.966559+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:46.966559+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9106, 'log_dups_size': 0, 'ondisk_log_size': 9106, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26591232, 'num_objects': 3246, 'num_object_clones': 0, 'num_object_copies': 12984, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3246, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9106, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 970}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9153", 'reported_seq': 9238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:48.564997+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:48.564997+0000', 'last_peered': '2024-09-15T21:10:48.564997+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:48.564997+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26860190, 'num_objects': 3280, 'num_object_clones': 0, 'num_object_copies': 13120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3280, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9152, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1022}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9321", 'reported_seq': 9395, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.305157+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:51.305157+0000', 'last_peered': '2024-09-15T21:10:51.305157+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:51.305157+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9321, 'log_dups_size': 0, 'ondisk_log_size': 9321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25679116, 'num_objects': 3135, 'num_object_clones': 0, 'num_object_copies': 12540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9319, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:52.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:52.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9353", 'reported_seq': 9414, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:46.943946+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:46.943946+0000', 'last_peered': '2024-09-15T21:10:46.943946+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:46.943946+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9353, 'log_dups_size': 0, 'ondisk_log_size': 9353, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27079948, 'num_objects': 3306, 'num_object_clones': 0, 'num_object_copies': 13224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9351, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 995}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9182", 'reported_seq': 9232, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:47.112626+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:47.112626+0000', 'last_peered': '2024-09-15T21:10:47.112626+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:47.112626+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9182, 'log_dups_size': 0, 'ondisk_log_size': 9182, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26165248, 'num_objects': 3194, 'num_object_clones': 0, 'num_object_copies': 12776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9182, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 878}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9493", 'reported_seq': 9568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.177145+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:51.177145+0000', 'last_peered': '2024-09-15T21:10:51.177145+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:51.177145+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9493, 'log_dups_size': 0, 'ondisk_log_size': 9493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26858764, 'num_objects': 3279, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9491, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 913}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9575", 'reported_seq': 9649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.228316+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:51.228316+0000', 'last_peered': '2024-09-15T21:10:51.228316+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:51.228316+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9575, 'log_dups_size': 0, 'ondisk_log_size': 9575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26096908, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9573, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:52.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9240", 'reported_seq': 9305, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:47.031216+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:47.031216+0000', 'last_peered': '2024-09-15T21:10:47.031216+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:47.031216+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9240, 'log_dups_size': 0, 'ondisk_log_size': 9240, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26307096, 'num_objects': 3212, 'num_object_clones': 0, 'num_object_copies': 12848, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3212, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9237, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 880}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9347", 'reported_seq': 9433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:48.906366+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:48.906366+0000', 'last_peered': '2024-09-15T21:10:48.906366+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:48.906366+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9347, 'log_dups_size': 0, 'ondisk_log_size': 9347, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26334476, 'num_objects': 3215, 'num_object_clones': 0, 'num_object_copies': 12860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9345, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 880}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9153", 'reported_seq': 9198, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:46.969254+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:46.969254+0000', 'last_peered': '2024-09-15T21:10:46.969254+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:46.969254+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9153, 'log_dups_size': 0, 'ondisk_log_size': 9153, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26819206, 'num_objects': 3274, 'num_object_clones': 0, 'num_object_copies': 13096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3274, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9152, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 1018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9252", 'reported_seq': 9548, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:48.888111+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:10:48.888111+0000', 'last_peered': '2024-09-15T21:10:48.888111+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:48.888111+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9252, 'log_dups_size': 0, 'ondisk_log_size': 9252, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26614406, 'num_objects': 3249, 'num_object_clones': 0, 'num_object_copies': 12996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2182, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9251, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 106, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1067}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9216", 'reported_seq': 9278, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:46.993980+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:46.993980+0000', 'last_peered': '2024-09-15T21:10:46.993980+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:46.993980+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9216, 'log_dups_size': 0, 'ondisk_log_size': 9216, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26342668, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9214, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 937}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:52.545 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9359", 'reported_seq': 9447, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:48.904424+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:48.904424+0000', 'last_peered': '2024-09-15T21:10:48.904424+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:48.904424+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9359, 'log_dups_size': 0, 'ondisk_log_size': 9359, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27211020, 'num_objects': 3322, 'num_object_clones': 0, 'num_object_copies': 13288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9357, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1041}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:52.546 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:52.546 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9354", 'reported_seq': 9428, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.206952+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:51.206952+0000', 'last_peered': '2024-09-15T21:10:51.206952+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:51.206952+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9354, 'log_dups_size': 0, 'ondisk_log_size': 9354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26170636, 'num_objects': 3195, 'num_object_clones': 0, 'num_object_copies': 12780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9352, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:52.546 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:53.274 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:53.275 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32336/206904 objects degraded (15.629%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":519,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":51726,"data_bytes":423698978,"bytes_used":2778595328,"bytes_avail":383768461312,"bytes_total":386547056640,"degraded_objects":32336,"degraded_total":206904,"degraded_ratio":0.15628504040521207},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:53.299 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:10:53.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:53.872 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:53.873 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:53.902 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-15T21:10:52.091107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":423698978,"num_objects":51726,"num_object_clones":0,"num_object_copies":206904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133352,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148597,"ondisk_log_size":148597,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2713472,"kb_used_data":828340,"kb_used_omap":14,"kb_used_meta":1885041,"kb_avail":374773888,"statfs":{"total":386547056640,"available":383768461312,"internally_reserved":0,"allocated":848220160,"data_stored":847527752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1930282546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,9,3,11,12,7,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16717068,"num_objects":-2041,"num_object_clones":0,"num_object_copies":-8164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7293,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2211,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.652829"},"pg_stats":[{"pgid":"1.f","version":"35'9106","reported_seq":9157,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.966559+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:46.966559+0000","last_peered":"2024-09-15T21:10:46.966559+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:46.966559+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9106,"log_dups_size":0,"ondisk_log_size":9106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26591232,"num_objects":3246,"num_object_clones":0,"num_object_copies":12984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9106,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":970}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9395","reported_seq":14288,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:50.072601+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:10:50.072601+0000","last_peered":"2024-09-15T21:10:50.072601+0000","last_clean":"2024-09-15T21:10:50.072601+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:50.072601+0000","last_undegraded":"2024-09-15T21:10:50.072601+0000","last_fullsized":"2024-09-15T21:10:50.072601+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9395,"log_dups_size":0,"ondisk_log_size":9395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25845760,"num_objects":3155,"num_object_clones":0,"num_object_copies":12620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9153","reported_seq":9238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.564997+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:48.564997+0000","last_peered":"2024-09-15T21:10:48.564997+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:48.564997+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26860190,"num_objects":3280,"num_object_clones":0,"num_object_copies":13120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9152,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1022}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9321","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.305157+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:51.305157+0000","last_peered":"2024-09-15T21:10:51.305157+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:51.305157+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9353","reported_seq":9414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.943946+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:46.943946+0000","last_peered":"2024-09-15T21:10:46.943946+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:46.943946+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9353,"log_dups_size":0,"ondisk_log_size":9353,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27079948,"num_objects":3306,"num_object_clones":0,"num_object_copies":13224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9182","reported_seq":9232,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.112626+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:47.112626+0000","last_peered":"2024-09-15T21:10:47.112626+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:47.112626+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9182,"log_dups_size":0,"ondisk_log_size":9182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26165248,"num_objects":3194,"num_object_clones":0,"num_object_copies":12776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9182,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":878}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9493","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.177145+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:51.177145+0000","last_peered":"2024-09-15T21:10:51.177145+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:51.177145+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858764,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9575","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.228316+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:51.228316+0000","last_peered":"2024-09-15T21:10:51.228316+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:51.228316+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096908,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9240","reported_seq":9305,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:47.031216+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:47.031216+0000","last_peered":"2024-09-15T21:10:47.031216+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:47.031216+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9240,"log_dups_size":0,"ondisk_log_size":9240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26307096,"num_objects":3212,"num_object_clones":0,"num_object_copies":12848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":880}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9347","reported_seq":9433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.906366+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:48.906366+0000","last_peered":"2024-09-15T21:10:48.906366+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:48.906366+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9347,"log_dups_size":0,"ondisk_log_size":9347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26334476,"num_objects":3215,"num_object_clones":0,"num_object_copies":12860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9345,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":880}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9098","reported_seq":13766,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:47.106908+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:47.106908+0000","last_peered":"2024-09-15T21:10:47.106908+0000","last_clean":"2024-09-15T21:10:47.106908+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:47.106908+0000","last_undegraded":"2024-09-15T21:10:47.106908+0000","last_fullsized":"2024-09-15T21:10:47.106908+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9098,"log_dups_size":0,"ondisk_log_size":9098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26722304,"num_objects":3262,"num_object_clones":0,"num_object_copies":13048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3209,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9153","reported_seq":9198,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.969254+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:46.969254+0000","last_peered":"2024-09-15T21:10:46.969254+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:46.969254+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26819206,"num_objects":3274,"num_object_clones":0,"num_object_copies":13096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":1018}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9252","reported_seq":9548,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.888111+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:48.888111+0000","last_peered":"2024-09-15T21:10:48.888111+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:48.888111+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9252,"log_dups_size":0,"ondisk_log_size":9252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26614406,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9251,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1067}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9216","reported_seq":9278,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:46.993980+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:46.993980+0000","last_peered":"2024-09-15T21:10:46.993980+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:46.993980+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9216,"log_dups_size":0,"ondisk_log_size":9216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26342668,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9214,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":937}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9359","reported_seq":9447,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:48.904424+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:48.904424+0000","last_peered":"2024-09-15T21:10:48.904424+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:48.904424+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27211020,"num_objects":3322,"num_object_clones":0,"num_object_copies":13288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9357,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1041}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9354","reported_seq":9428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.206952+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:51.206952+0000","last_peered":"2024-09-15T21:10:51.206952+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:51.206952+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26170636,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":423698978,"num_objects":51726,"num_object_clones":0,"num_object_copies":206904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51726,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133352,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":845393920,"data_stored":845393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":148597,"ondisk_log_size":148597,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":718324,"kb_used_data":209060,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93653516,"statfs":{"total":96636764160,"available":95901200384,"internally_reserved":0,"allocated":214077440,"data_stored":213904338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,8,2,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707852,"kb_used_data":207416,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93663988,"statfs":{"total":96636764160,"available":95911923712,"internally_reserved":0,"allocated":212393984,"data_stored":212220882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051667,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577008,"kb_used_data":206620,"kb_used_omap":4,"kb_used_meta":370363,"kb_avail":93794832,"statfs":{"total":96636764160,"available":96045907968,"internally_reserved":0,"allocated":211578880,"data_stored":211405778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":379252431},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710288,"kb_used_data":205244,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93661552,"statfs":{"total":96636764160,"available":95909429248,"internally_reserved":0,"allocated":210169856,"data_stored":209996754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,1,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209833984,"data_stored":209833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":210661376,"data_stored":210661376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":211705856,"data_stored":211705856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":213192704,"data_stored":213192704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:56.904 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:57.445 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:57.446 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:57.467 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-15T21:10:56.092839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417718818,"num_objects":50996,"num_object_clones":0,"num_object_copies":203984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134082,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149327,"ondisk_log_size":149327,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733676,"kb_used_data":816600,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374753684,"statfs":{"total":386547056640,"available":383747772416,"internally_reserved":0,"allocated":836198400,"data_stored":835505992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1962985010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,13,2,3,11,17,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17159436,"num_objects":-2095,"num_object_clones":0,"num_object_copies":-8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7239,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2213,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007761"},"pg_stats":[{"pgid":"1.f","version":"35'9167","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.111793+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:52.111793+0000","last_peered":"2024-09-15T21:10:52.111793+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:52.111793+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26091520,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9461","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:55.502921+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:10:55.502921+0000","last_peered":"2024-09-15T21:10:55.502921+0000","last_clean":"2024-09-15T21:10:55.502921+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:55.502921+0000","last_undegraded":"2024-09-15T21:10:55.502921+0000","last_fullsized":"2024-09-15T21:10:55.502921+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25305088,"num_objects":3089,"num_object_clones":0,"num_object_copies":12356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9213","reported_seq":9298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.854892+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:53.854892+0000","last_peered":"2024-09-15T21:10:53.854892+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:53.854892+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26368670,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9212,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9321","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.305157+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:51.305157+0000","last_peered":"2024-09-15T21:10:51.305157+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:51.305157+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9407","reported_seq":9468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.215338+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:52.215338+0000","last_peered":"2024-09-15T21:10:52.215338+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:52.215338+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26637580,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9234","reported_seq":9284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.122244+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:52.122244+0000","last_peered":"2024-09-15T21:10:52.122244+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:52.122244+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25739264,"num_objects":3142,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9493","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.177145+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:51.177145+0000","last_peered":"2024-09-15T21:10:51.177145+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:51.177145+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858764,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9575","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.228316+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:51.228316+0000","last_peered":"2024-09-15T21:10:51.228316+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:51.228316+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096908,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9301","reported_seq":9366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.147992+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:52.147992+0000","last_peered":"2024-09-15T21:10:52.147992+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:52.147992+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25807384,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9408","reported_seq":9494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.938385+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:53.938385+0000","last_peered":"2024-09-15T21:10:53.938385+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:53.938385+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25834764,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9406,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9155","reported_seq":13823,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:52.185388+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:52.185388+0000","last_peered":"2024-09-15T21:10:52.185388+0000","last_clean":"2024-09-15T21:10:52.185388+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:52.185388+0000","last_undegraded":"2024-09-15T21:10:52.185388+0000","last_fullsized":"2024-09-15T21:10:52.185388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26255360,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9211","reported_seq":9256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.102709+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:52.102709+0000","last_peered":"2024-09-15T21:10:52.102709+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:52.102709+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26344070,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9210,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9314","reported_seq":9706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.930842+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:53.930842+0000","last_peered":"2024-09-15T21:10:53.930842+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:53.930842+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26106502,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9281","reported_seq":9343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.138376+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:52.138376+0000","last_peered":"2024-09-15T21:10:52.138376+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:52.138376+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25810188,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9432","reported_seq":9520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.919725+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:53.919725+0000","last_peered":"2024-09-15T21:10:53.919725+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:53.919725+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26613004,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9354","reported_seq":9428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.206952+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:51.206952+0000","last_peered":"2024-09-15T21:10:51.206952+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:51.206952+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26170636,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417718818,"num_objects":50996,"num_object_clones":0,"num_object_copies":203984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134082,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":833175552,"data_stored":833175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149327,"ondisk_log_size":149327,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701956,"kb_used_data":205172,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93669884,"statfs":{"total":96636764160,"available":95917961216,"internally_reserved":0,"allocated":210096128,"data_stored":209923026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,1,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708092,"kb_used_data":203500,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93663748,"statfs":{"total":96636764160,"available":95911677952,"internally_reserved":0,"allocated":208384000,"data_stored":208210898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613340,"kb_used_data":202684,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93758500,"statfs":{"total":96636764160,"available":96008704000,"internally_reserved":0,"allocated":207548416,"data_stored":207375314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710288,"kb_used_data":205244,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93661552,"statfs":{"total":96636764160,"available":95909429248,"internally_reserved":0,"allocated":210169856,"data_stored":209996754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,1,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209833984,"data_stored":209833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206483456,"data_stored":206483456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209121280,"data_stored":209121280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:57.469 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:58.057 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:58.057 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:58.100 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-15T21:10:56.092839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417718818,"num_objects":50996,"num_object_clones":0,"num_object_copies":203984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134082,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149327,"ondisk_log_size":149327,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733676,"kb_used_data":816600,"kb_used_omap":14,"kb_used_meta":1916977,"kb_avail":374753684,"statfs":{"total":386547056640,"available":383747772416,"internally_reserved":0,"allocated":836198400,"data_stored":835505992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1962985010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,13,2,3,11,17,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17159436,"num_objects":-2095,"num_object_clones":0,"num_object_copies":-8380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-171,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-7239,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2213,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007761"},"pg_stats":[{"pgid":"1.f","version":"35'9167","reported_seq":9218,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.111793+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:52.111793+0000","last_peered":"2024-09-15T21:10:52.111793+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:52.111793+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9167,"log_dups_size":0,"ondisk_log_size":9167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26091520,"num_objects":3185,"num_object_clones":0,"num_object_copies":12740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9167,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":909}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9461","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:55.502921+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:10:55.502921+0000","last_peered":"2024-09-15T21:10:55.502921+0000","last_clean":"2024-09-15T21:10:55.502921+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:55.502921+0000","last_undegraded":"2024-09-15T21:10:55.502921+0000","last_fullsized":"2024-09-15T21:10:55.502921+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25305088,"num_objects":3089,"num_object_clones":0,"num_object_copies":12356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9213","reported_seq":9298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.854892+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:53.854892+0000","last_peered":"2024-09-15T21:10:53.854892+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:53.854892+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26368670,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9212,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9321","reported_seq":9395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.305157+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:51.305157+0000","last_peered":"2024-09-15T21:10:51.305157+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:51.305157+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9321,"log_dups_size":0,"ondisk_log_size":9321,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25679116,"num_objects":3135,"num_object_clones":0,"num_object_copies":12540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":801}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9407","reported_seq":9468,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.215338+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:52.215338+0000","last_peered":"2024-09-15T21:10:52.215338+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:52.215338+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9407,"log_dups_size":0,"ondisk_log_size":9407,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26637580,"num_objects":3252,"num_object_clones":0,"num_object_copies":13008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":941}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9234","reported_seq":9284,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.122244+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:52.122244+0000","last_peered":"2024-09-15T21:10:52.122244+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:52.122244+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9234,"log_dups_size":0,"ondisk_log_size":9234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25739264,"num_objects":3142,"num_object_clones":0,"num_object_copies":12568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9493","reported_seq":9568,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.177145+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:51.177145+0000","last_peered":"2024-09-15T21:10:51.177145+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:51.177145+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9493,"log_dups_size":0,"ondisk_log_size":9493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26858764,"num_objects":3279,"num_object_clones":0,"num_object_copies":13116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9491,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":913}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9575","reported_seq":9649,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.228316+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:51.228316+0000","last_peered":"2024-09-15T21:10:51.228316+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:51.228316+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9575,"log_dups_size":0,"ondisk_log_size":9575,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26096908,"num_objects":3186,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9573,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9301","reported_seq":9366,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.147992+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:52.147992+0000","last_peered":"2024-09-15T21:10:52.147992+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:52.147992+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25807384,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":819}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9408","reported_seq":9494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.938385+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:53.938385+0000","last_peered":"2024-09-15T21:10:53.938385+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:53.938385+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25834764,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9406,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9155","reported_seq":13823,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:52.185388+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:52.185388+0000","last_peered":"2024-09-15T21:10:52.185388+0000","last_clean":"2024-09-15T21:10:52.185388+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:52.185388+0000","last_undegraded":"2024-09-15T21:10:52.185388+0000","last_fullsized":"2024-09-15T21:10:52.185388+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9155,"log_dups_size":0,"ondisk_log_size":9155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26255360,"num_objects":3205,"num_object_clones":0,"num_object_copies":12820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9211","reported_seq":9256,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.102709+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:52.102709+0000","last_peered":"2024-09-15T21:10:52.102709+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:52.102709+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9211,"log_dups_size":0,"ondisk_log_size":9211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26344070,"num_objects":3216,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9210,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9314","reported_seq":9706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.930842+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:53.930842+0000","last_peered":"2024-09-15T21:10:53.930842+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:53.930842+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26106502,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9281","reported_seq":9343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:52.138376+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:52.138376+0000","last_peered":"2024-09-15T21:10:52.138376+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:52.138376+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9281,"log_dups_size":0,"ondisk_log_size":9281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25810188,"num_objects":3151,"num_object_clones":0,"num_object_copies":12604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9279,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":872}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9432","reported_seq":9520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.919725+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:53.919725+0000","last_peered":"2024-09-15T21:10:53.919725+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:53.919725+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26613004,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9354","reported_seq":9428,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:51.206952+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:51.206952+0000","last_peered":"2024-09-15T21:10:51.206952+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:51.206952+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26170636,"num_objects":3195,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":417718818,"num_objects":50996,"num_object_clones":0,"num_object_copies":203984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50996,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134082,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":833175552,"data_stored":833175552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149327,"ondisk_log_size":149327,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701956,"kb_used_data":205172,"kb_used_omap":4,"kb_used_meta":496763,"kb_avail":93669884,"statfs":{"total":96636764160,"available":95917961216,"internally_reserved":0,"allocated":210096128,"data_stored":209923026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":508686032},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,8,1,1,5,6,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708092,"kb_used_data":203500,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93663748,"statfs":{"total":96636764160,"available":95911677952,"internally_reserved":0,"allocated":208384000,"data_stored":208210898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613340,"kb_used_data":202684,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93758500,"statfs":{"total":96636764160,"available":96008704000,"internally_reserved":0,"allocated":207548416,"data_stored":207375314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247905,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710288,"kb_used_data":205244,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93661552,"statfs":{"total":96636764160,"available":95909429248,"internally_reserved":0,"allocated":210169856,"data_stored":209996754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,1,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":209833984,"data_stored":209833984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206483456,"data_stored":206483456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":209121280,"data_stored":209121280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9167", 'reported_seq': 9218, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.111793+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:52.111793+0000', 'last_peered': '2024-09-15T21:10:52.111793+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:52.111793+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9167, 'log_dups_size': 0, 'ondisk_log_size': 9167, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26091520, 'num_objects': 3185, 'num_object_clones': 0, 'num_object_copies': 12740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9167, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 909}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9213", 'reported_seq': 9298, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:53.854892+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:53.854892+0000', 'last_peered': '2024-09-15T21:10:53.854892+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:53.854892+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9213, 'log_dups_size': 0, 'ondisk_log_size': 9213, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26368670, 'num_objects': 3220, 'num_object_clones': 0, 'num_object_copies': 12880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3220, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9212, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 962}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:10:58.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9321", 'reported_seq': 9395, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.305157+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:10:51.305157+0000', 'last_peered': '2024-09-15T21:10:51.305157+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:10:51.305157+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9321, 'log_dups_size': 0, 'ondisk_log_size': 9321, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25679116, 'num_objects': 3135, 'num_object_clones': 0, 'num_object_copies': 12540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9319, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 801}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9407", 'reported_seq': 9468, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.215338+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:52.215338+0000', 'last_peered': '2024-09-15T21:10:52.215338+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:52.215338+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9407, 'log_dups_size': 0, 'ondisk_log_size': 9407, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26637580, 'num_objects': 3252, 'num_object_clones': 0, 'num_object_copies': 13008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9405, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 941}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9234", 'reported_seq': 9284, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.122244+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:52.122244+0000', 'last_peered': '2024-09-15T21:10:52.122244+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:52.122244+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9234, 'log_dups_size': 0, 'ondisk_log_size': 9234, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25739264, 'num_objects': 3142, 'num_object_clones': 0, 'num_object_copies': 12568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3142, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9234, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9493", 'reported_seq': 9568, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.177145+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:10:51.177145+0000', 'last_peered': '2024-09-15T21:10:51.177145+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:10:51.177145+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9493, 'log_dups_size': 0, 'ondisk_log_size': 9493, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26858764, 'num_objects': 3279, 'num_object_clones': 0, 'num_object_copies': 13116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9491, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 913}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9575", 'reported_seq': 9649, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.228316+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:10:51.228316+0000', 'last_peered': '2024-09-15T21:10:51.228316+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:10:51.228316+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9575, 'log_dups_size': 0, 'ondisk_log_size': 9575, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26096908, 'num_objects': 3186, 'num_object_clones': 0, 'num_object_copies': 12744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9573, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9301", 'reported_seq': 9366, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.147992+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:52.147992+0000', 'last_peered': '2024-09-15T21:10:52.147992+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:52.147992+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9301, 'log_dups_size': 0, 'ondisk_log_size': 9301, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25807384, 'num_objects': 3151, 'num_object_clones': 0, 'num_object_copies': 12604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9298, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 819}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9408", 'reported_seq': 9494, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:53.938385+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:53.938385+0000', 'last_peered': '2024-09-15T21:10:53.938385+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:53.938385+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9408, 'log_dups_size': 0, 'ondisk_log_size': 9408, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25834764, 'num_objects': 3154, 'num_object_clones': 0, 'num_object_copies': 12616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3154, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9406, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 819}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9211", 'reported_seq': 9256, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.102709+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:52.102709+0000', 'last_peered': '2024-09-15T21:10:52.102709+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:52.102709+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9211, 'log_dups_size': 0, 'ondisk_log_size': 9211, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26344070, 'num_objects': 3216, 'num_object_clones': 0, 'num_object_copies': 12864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3216, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9210, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9314", 'reported_seq': 9706, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:53.930842+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:10:53.930842+0000', 'last_peered': '2024-09-15T21:10:53.930842+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:53.930842+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9314, 'log_dups_size': 0, 'ondisk_log_size': 9314, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26106502, 'num_objects': 3187, 'num_object_clones': 0, 'num_object_copies': 12748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2134, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9313, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 154, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1053}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9281", 'reported_seq': 9343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:52.138376+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:52.138376+0000', 'last_peered': '2024-09-15T21:10:52.138376+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:52.138376+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9281, 'log_dups_size': 0, 'ondisk_log_size': 9281, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25810188, 'num_objects': 3151, 'num_object_clones': 0, 'num_object_copies': 12604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9279, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 872}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9432", 'reported_seq': 9520, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:53.919725+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:53.919725+0000', 'last_peered': '2024-09-15T21:10:53.919725+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:53.919725+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9432, 'log_dups_size': 0, 'ondisk_log_size': 9432, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26613004, 'num_objects': 3249, 'num_object_clones': 0, 'num_object_copies': 12996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9430, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 968}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:10:58.103 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9354", 'reported_seq': 9428, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:51.206952+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:10:51.206952+0000', 'last_peered': '2024-09-15T21:10:51.206952+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:10:51.206952+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9354, 'log_dups_size': 0, 'ondisk_log_size': 9354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26170636, 'num_objects': 3195, 'num_object_clones': 0, 'num_object_copies': 12780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9352, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:10:58.104 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:10:58.923 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:58.923 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32288/200980 objects degraded (16.065%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":524,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":50245,"data_bytes":411566626,"bytes_used":2802946048,"bytes_avail":383744110592,"bytes_total":386547056640,"degraded_objects":32288,"degraded_total":200980,"degraded_ratio":0.16065280127375858},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:10:58.943 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:10:58.944 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:10:59.552 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:10:59.552 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:10:59.605 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2024-09-15T21:10:58.093594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":411566626,"num_objects":50245,"num_object_clones":0,"num_object_copies":200980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134833,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150078,"ondisk_log_size":150078,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737252,"kb_used_data":807680,"kb_used_omap":14,"kb_used_meta":1929457,"kb_avail":374750108,"statfs":{"total":386547056640,"available":383744110592,"internally_reserved":0,"allocated":827064320,"data_stored":826371912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1975764530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,6,2,10,13,12,14,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22304012,"num_objects":-2723,"num_object_clones":0,"num_object_copies":-10892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6611,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2281,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007572"},"pg_stats":[{"pgid":"1.f","version":"35'9224","reported_seq":9274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.364550+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:57.364550+0000","last_peered":"2024-09-15T21:10:57.364550+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:57.364550+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9461","reported_seq":14354,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:55.502921+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:10:55.502921+0000","last_peered":"2024-09-15T21:10:55.502921+0000","last_clean":"2024-09-15T21:10:55.502921+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:10:55.502921+0000","last_undegraded":"2024-09-15T21:10:55.502921+0000","last_fullsized":"2024-09-15T21:10:55.502921+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9461,"log_dups_size":0,"ondisk_log_size":9461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25305088,"num_objects":3089,"num_object_clones":0,"num_object_copies":12356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":127,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9213","reported_seq":9298,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.854892+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:53.854892+0000","last_peered":"2024-09-15T21:10:53.854892+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:53.854892+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9213,"log_dups_size":0,"ondisk_log_size":9213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26368670,"num_objects":3220,"num_object_clones":0,"num_object_copies":12880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9212,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":962}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9391","reported_seq":9465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:56.239334+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:10:56.239334+0000","last_peered":"2024-09-15T21:10:56.239334+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:10:56.239334+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9391,"log_dups_size":0,"ondisk_log_size":9391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25105676,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9389,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":731}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9463","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.259591+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:57.259591+0000","last_peered":"2024-09-15T21:10:57.259591+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:57.259591+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178828,"num_objects":3196,"num_object_clones":0,"num_object_copies":12784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9307","reported_seq":9357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.421546+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:57.421546+0000","last_peered":"2024-09-15T21:10:57.421546+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:57.421546+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9558","reported_seq":9633,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:56.364402+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:10:56.364402+0000","last_peered":"2024-09-15T21:10:56.364402+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:10:56.364402+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9558,"log_dups_size":0,"ondisk_log_size":9558,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26326284,"num_objects":3214,"num_object_clones":0,"num_object_copies":12856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9556,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":848}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9650","reported_seq":9723,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:56.374055+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:10:56.374055+0000","last_peered":"2024-09-15T21:10:56.374055+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:10:56.374055+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9650,"log_dups_size":0,"ondisk_log_size":9650,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25482508,"num_objects":3111,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9648,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":637}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9354","reported_seq":9419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.190480+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:57.190480+0000","last_peered":"2024-09-15T21:10:57.190480+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:57.190480+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25373208,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9408","reported_seq":9494,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.938385+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:53.938385+0000","last_peered":"2024-09-15T21:10:53.938385+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:53.938385+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9408,"log_dups_size":0,"ondisk_log_size":9408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25834764,"num_objects":3154,"num_object_clones":0,"num_object_copies":12616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9406,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":819}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9231","reported_seq":13899,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:57.254674+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:57.254674+0000","last_peered":"2024-09-15T21:10:57.254674+0000","last_clean":"2024-09-15T21:10:57.254674+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:57.254674+0000","last_undegraded":"2024-09-15T21:10:57.254674+0000","last_fullsized":"2024-09-15T21:10:57.254674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25632768,"num_objects":3129,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9278","reported_seq":9323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.224068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:57.224068+0000","last_peered":"2024-09-15T21:10:57.224068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:57.224068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25795206,"num_objects":3149,"num_object_clones":0,"num_object_copies":12596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9314","reported_seq":9706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.930842+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:53.930842+0000","last_peered":"2024-09-15T21:10:53.930842+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:53.930842+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9314,"log_dups_size":0,"ondisk_log_size":9314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26106502,"num_objects":3187,"num_object_clones":0,"num_object_copies":12748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2134,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9313,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1053}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9349","reported_seq":9411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.339853+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:57.339853+0000","last_peered":"2024-09-15T21:10:57.339853+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:57.339853+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253132,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9432","reported_seq":9520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:53.919725+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:53.919725+0000","last_peered":"2024-09-15T21:10:53.919725+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:53.919725+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26613004,"num_objects":3249,"num_object_clones":0,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9430,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":968}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9445","reported_seq":9519,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:56.355730+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:10:56.355730+0000","last_peered":"2024-09-15T21:10:56.355730+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:10:56.355730+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9445,"log_dups_size":0,"ondisk_log_size":9445,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25425164,"num_objects":3104,"num_object_clones":0,"num_object_copies":12416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9443,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":789}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":411566626,"num_objects":50245,"num_object_clones":0,"num_object_copies":200980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134833,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":824582144,"data_stored":824582144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150078,"ondisk_log_size":150078,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705480,"kb_used_data":200356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666360,"statfs":{"total":96636764160,"available":95914352640,"internally_reserved":0,"allocated":205164544,"data_stored":204991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,7,1,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708092,"kb_used_data":203500,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93663748,"statfs":{"total":96636764160,"available":95911677952,"internally_reserved":0,"allocated":208384000,"data_stored":208210898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051668,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613340,"kb_used_data":202684,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93758500,"statfs":{"total":96636764160,"available":96008704000,"internally_reserved":0,"allocated":207548416,"data_stored":207375314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,2,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247906,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710340,"kb_used_data":201140,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93661500,"statfs":{"total":96636764160,"available":95909376000,"internally_reserved":0,"allocated":205967360,"data_stored":205794258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":521400014},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,1,0,1,0,3,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205586432,"data_stored":205586432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206483456,"data_stored":206483456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":207736832,"data_stored":207736832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204775424,"data_stored":204775424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:02.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:03.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:03.213 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:03.248 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-15T21:11:02.095253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407028258,"num_objects":49691,"num_object_clones":0,"num_object_copies":198764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135387,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150632,"ondisk_log_size":150632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721568,"kb_used_data":796156,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374765792,"statfs":{"total":386547056640,"available":383760171008,"internally_reserved":0,"allocated":815263744,"data_stored":814571336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1971504690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,16,22,12,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19207436,"num_objects":-2345,"num_object_clones":0,"num_object_copies":-9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6989,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2283,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"1.f","version":"35'9224","reported_seq":9274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.364550+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:57.364550+0000","last_peered":"2024-09-15T21:10:57.364550+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:57.364550+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9503","reported_seq":14395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:00.475524+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:00.475524+0000","last_peered":"2024-09-15T21:11:00.475524+0000","last_clean":"2024-09-15T21:11:00.475524+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:00.475524+0000","last_undegraded":"2024-09-15T21:11:00.475524+0000","last_fullsized":"2024-09-15T21:11:00.475524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9271","reported_seq":9356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.984168+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:58.984168+0000","last_peered":"2024-09-15T21:10:58.984168+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:58.984168+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25893534,"num_objects":3162,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9270,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9449","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.235201+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:01.235201+0000","last_peered":"2024-09-15T21:11:01.235201+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:01.235201+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24630540,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9463","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.259591+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:57.259591+0000","last_peered":"2024-09-15T21:10:57.259591+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:57.259591+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178828,"num_objects":3196,"num_object_clones":0,"num_object_copies":12784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9307","reported_seq":9357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.421546+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:57.421546+0000","last_peered":"2024-09-15T21:10:57.421546+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:57.421546+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9626","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.321220+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:01.321220+0000","last_peered":"2024-09-15T21:11:01.321220+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:01.321220+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":780}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9706","reported_seq":9780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.331468+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:01.331468+0000","last_peered":"2024-09-15T21:11:01.331468+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:01.331468+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9354","reported_seq":9419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.190480+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:57.190480+0000","last_peered":"2024-09-15T21:10:57.190480+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:57.190480+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25373208,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9480","reported_seq":9566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.088217+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:59.088217+0000","last_peered":"2024-09-15T21:10:59.088217+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:59.088217+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25244940,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":747}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9231","reported_seq":13899,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:57.254674+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:57.254674+0000","last_peered":"2024-09-15T21:10:57.254674+0000","last_clean":"2024-09-15T21:10:57.254674+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:57.254674+0000","last_undegraded":"2024-09-15T21:10:57.254674+0000","last_fullsized":"2024-09-15T21:10:57.254674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25632768,"num_objects":3129,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9278","reported_seq":9323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.224068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:57.224068+0000","last_peered":"2024-09-15T21:10:57.224068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:57.224068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25795206,"num_objects":3149,"num_object_clones":0,"num_object_copies":12596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9384","reported_seq":9882,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.043937+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:59.043937+0000","last_peered":"2024-09-15T21:10:59.043937+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:59.043937+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9384,"log_dups_size":0,"ondisk_log_size":9384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533062,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9349","reported_seq":9411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.339853+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:57.339853+0000","last_peered":"2024-09-15T21:10:57.339853+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:57.339853+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253132,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9503","reported_seq":9591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.845188+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:58.845188+0000","last_peered":"2024-09-15T21:10:58.845188+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:58.845188+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26031372,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9504","reported_seq":9578,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:00.989916+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:00.989916+0000","last_peered":"2024-09-15T21:11:00.989916+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:00.989916+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24941836,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407028258,"num_objects":49691,"num_object_clones":0,"num_object_copies":198764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135387,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811999232,"data_stored":811999232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150632,"ondisk_log_size":150632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705480,"kb_used_data":200356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666360,"statfs":{"total":96636764160,"available":95914352640,"internally_reserved":0,"allocated":205164544,"data_stored":204991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,7,1,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708076,"kb_used_data":199308,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93663764,"statfs":{"total":96636764160,"available":95911694336,"internally_reserved":0,"allocated":204091392,"data_stored":203918290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":198628,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":203395072,"data_stored":203221970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694584,"kb_used_data":197864,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93677256,"statfs":{"total":96636764160,"available":95925510144,"internally_reserved":0,"allocated":202612736,"data_stored":202439634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201461760,"data_stored":201461760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202321920,"data_stored":202321920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203440128,"data_stored":203440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204775424,"data_stored":204775424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:03.250 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:03.894 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:03.895 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:03.931 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-15T21:11:02.095253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":407028258,"num_objects":49691,"num_object_clones":0,"num_object_copies":198764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135387,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150632,"ondisk_log_size":150632,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721568,"kb_used_data":796156,"kb_used_omap":14,"kb_used_meta":1925297,"kb_avail":374765792,"statfs":{"total":386547056640,"available":383760171008,"internally_reserved":0,"allocated":815263744,"data_stored":814571336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1971504690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,16,22,12,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19207436,"num_objects":-2345,"num_object_clones":0,"num_object_copies":-9380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-6989,"num_write_kb":-50204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2283,"num_bytes_recovered":-13950976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004782"},"pg_stats":[{"pgid":"1.f","version":"35'9224","reported_seq":9274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.364550+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:10:57.364550+0000","last_peered":"2024-09-15T21:10:57.364550+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:10:57.364550+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9224,"log_dups_size":0,"ondisk_log_size":9224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25624576,"num_objects":3128,"num_object_clones":0,"num_object_copies":12512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":852}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9503","reported_seq":14395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:00.475524+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:00.475524+0000","last_peered":"2024-09-15T21:11:00.475524+0000","last_clean":"2024-09-15T21:11:00.475524+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:00.475524+0000","last_undegraded":"2024-09-15T21:11:00.475524+0000","last_fullsized":"2024-09-15T21:11:00.475524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9271","reported_seq":9356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.984168+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:58.984168+0000","last_peered":"2024-09-15T21:10:58.984168+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:58.984168+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25893534,"num_objects":3162,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9270,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9449","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.235201+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:01.235201+0000","last_peered":"2024-09-15T21:11:01.235201+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:01.235201+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24630540,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9463","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.259591+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:10:57.259591+0000","last_peered":"2024-09-15T21:10:57.259591+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:10:57.259591+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9463,"log_dups_size":0,"ondisk_log_size":9463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26178828,"num_objects":3196,"num_object_clones":0,"num_object_copies":12784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9461,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9307","reported_seq":9357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.421546+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:10:57.421546+0000","last_peered":"2024-09-15T21:10:57.421546+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:10:57.421546+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9307,"log_dups_size":0,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25141248,"num_objects":3069,"num_object_clones":0,"num_object_copies":12276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9307,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":753}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9626","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.321220+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:01.321220+0000","last_peered":"2024-09-15T21:11:01.321220+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:01.321220+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":780}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9706","reported_seq":9780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.331468+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:01.331468+0000","last_peered":"2024-09-15T21:11:01.331468+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:01.331468+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9354","reported_seq":9419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.190480+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:10:57.190480+0000","last_peered":"2024-09-15T21:10:57.190480+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:10:57.190480+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9354,"log_dups_size":0,"ondisk_log_size":9354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25373208,"num_objects":3098,"num_object_clones":0,"num_object_copies":12392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9480","reported_seq":9566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.088217+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:59.088217+0000","last_peered":"2024-09-15T21:10:59.088217+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:59.088217+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25244940,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":747}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9231","reported_seq":13899,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:10:57.254674+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:10:57.254674+0000","last_peered":"2024-09-15T21:10:57.254674+0000","last_clean":"2024-09-15T21:10:57.254674+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:10:57.254674+0000","last_undegraded":"2024-09-15T21:10:57.254674+0000","last_fullsized":"2024-09-15T21:10:57.254674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9231,"log_dups_size":0,"ondisk_log_size":9231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25632768,"num_objects":3129,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3342,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9278","reported_seq":9323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.224068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:10:57.224068+0000","last_peered":"2024-09-15T21:10:57.224068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:10:57.224068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9278,"log_dups_size":0,"ondisk_log_size":9278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25795206,"num_objects":3149,"num_object_clones":0,"num_object_copies":12596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9277,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":893}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9384","reported_seq":9882,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.043937+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:59.043937+0000","last_peered":"2024-09-15T21:10:59.043937+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:59.043937+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9384,"log_dups_size":0,"ondisk_log_size":9384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533062,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9349","reported_seq":9411,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:57.339853+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:10:57.339853+0000","last_peered":"2024-09-15T21:10:57.339853+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:10:57.339853+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9349,"log_dups_size":0,"ondisk_log_size":9349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25253132,"num_objects":3083,"num_object_clones":0,"num_object_copies":12332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9347,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":804}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9503","reported_seq":9591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.845188+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:58.845188+0000","last_peered":"2024-09-15T21:10:58.845188+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:58.845188+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26031372,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9504","reported_seq":9578,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:00.989916+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:00.989916+0000","last_peered":"2024-09-15T21:11:00.989916+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:00.989916+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24941836,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":407028258,"num_objects":49691,"num_object_clones":0,"num_object_copies":198764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135387,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811999232,"data_stored":811999232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150632,"ondisk_log_size":150632,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705480,"kb_used_data":200356,"kb_used_omap":4,"kb_used_meta":505083,"kb_avail":93666360,"statfs":{"total":96636764160,"available":95914352640,"internally_reserved":0,"allocated":205164544,"data_stored":204991442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":517205712},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,7,1,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708076,"kb_used_data":199308,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93663764,"statfs":{"total":96636764160,"available":95911694336,"internally_reserved":0,"allocated":204091392,"data_stored":203918290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":198628,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":203395072,"data_stored":203221970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694584,"kb_used_data":197864,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93677256,"statfs":{"total":96636764160,"available":95925510144,"internally_reserved":0,"allocated":202612736,"data_stored":202439634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201461760,"data_stored":201461760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202321920,"data_stored":202321920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203440128,"data_stored":203440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204775424,"data_stored":204775424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:03.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:03.932 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9224", 'reported_seq': 9274, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.364550+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:10:57.364550+0000', 'last_peered': '2024-09-15T21:10:57.364550+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:10:57.364550+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9224, 'log_dups_size': 0, 'ondisk_log_size': 9224, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25624576, 'num_objects': 3128, 'num_object_clones': 0, 'num_object_copies': 12512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9224, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 852}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.932 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9271", 'reported_seq': 9356, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:58.984168+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:10:58.984168+0000', 'last_peered': '2024-09-15T21:10:58.984168+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:10:58.984168+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9271, 'log_dups_size': 0, 'ondisk_log_size': 9271, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25893534, 'num_objects': 3162, 'num_object_clones': 0, 'num_object_copies': 12648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3162, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9270, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 904}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9449", 'reported_seq': 9523, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:01.235201+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:01.235201+0000', 'last_peered': '2024-09-15T21:11:01.235201+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:01.235201+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9449, 'log_dups_size': 0, 'ondisk_log_size': 9449, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24630540, 'num_objects': 3007, 'num_object_clones': 0, 'num_object_copies': 12028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9447, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 673}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9463", 'reported_seq': 9523, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.259591+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:10:57.259591+0000', 'last_peered': '2024-09-15T21:10:57.259591+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:10:57.259591+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9463, 'log_dups_size': 0, 'ondisk_log_size': 9463, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26178828, 'num_objects': 3196, 'num_object_clones': 0, 'num_object_copies': 12784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9461, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 885}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9307", 'reported_seq': 9357, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.421546+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:10:57.421546+0000', 'last_peered': '2024-09-15T21:10:57.421546+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:10:57.421546+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9307, 'log_dups_size': 0, 'ondisk_log_size': 9307, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25141248, 'num_objects': 3069, 'num_object_clones': 0, 'num_object_copies': 12276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9307, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 753}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:03.933 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9626", 'reported_seq': 9701, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:01.321220+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:01.321220+0000', 'last_peered': '2024-09-15T21:11:01.321220+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:01.321220+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9626, 'log_dups_size': 0, 'ondisk_log_size': 9626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25769228, 'num_objects': 3146, 'num_object_clones': 0, 'num_object_copies': 12584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3146, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9624, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 780}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9706", 'reported_seq': 9780, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:01.331468+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:01.331468+0000', 'last_peered': '2024-09-15T21:11:01.331468+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:01.331468+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9706, 'log_dups_size': 0, 'ondisk_log_size': 9706, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25023756, 'num_objects': 3055, 'num_object_clones': 0, 'num_object_copies': 12220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3055, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9704, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9354", 'reported_seq': 9419, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.190480+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:10:57.190480+0000', 'last_peered': '2024-09-15T21:10:57.190480+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:10:57.190480+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9354, 'log_dups_size': 0, 'ondisk_log_size': 9354, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25373208, 'num_objects': 3098, 'num_object_clones': 0, 'num_object_copies': 12392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3098, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9351, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9480", 'reported_seq': 9566, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:59.088217+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:10:59.088217+0000', 'last_peered': '2024-09-15T21:10:59.088217+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:10:59.088217+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9480, 'log_dups_size': 0, 'ondisk_log_size': 9480, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25244940, 'num_objects': 3082, 'num_object_clones': 0, 'num_object_copies': 12328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3082, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9478, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 747}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9278", 'reported_seq': 9323, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.224068+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:10:57.224068+0000', 'last_peered': '2024-09-15T21:10:57.224068+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:10:57.224068+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9278, 'log_dups_size': 0, 'ondisk_log_size': 9278, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25795206, 'num_objects': 3149, 'num_object_clones': 0, 'num_object_copies': 12596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9277, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 893}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9384", 'reported_seq': 9882, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:59.043937+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:10:59.043937+0000', 'last_peered': '2024-09-15T21:10:59.043937+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:10:59.043937+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9384, 'log_dups_size': 0, 'ondisk_log_size': 9384, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25533062, 'num_objects': 3117, 'num_object_clones': 0, 'num_object_copies': 12468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2081, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3117, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9383, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 207, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1036}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:03.934 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:03.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9349", 'reported_seq': 9411, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:57.339853+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:10:57.339853+0000', 'last_peered': '2024-09-15T21:10:57.339853+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:10:57.339853+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9349, 'log_dups_size': 0, 'ondisk_log_size': 9349, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25253132, 'num_objects': 3083, 'num_object_clones': 0, 'num_object_copies': 12332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3083, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9347, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 804}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:03.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:03.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9503", 'reported_seq': 9591, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:10:58.845188+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:10:58.845188+0000', 'last_peered': '2024-09-15T21:10:58.845188+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:10:58.845188+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9503, 'log_dups_size': 0, 'ondisk_log_size': 9503, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 26031372, 'num_objects': 3178, 'num_object_clones': 0, 'num_object_copies': 12712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3178, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9501, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:03.935 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:03.935 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9504", 'reported_seq': 9578, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:00.989916+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:00.989916+0000', 'last_peered': '2024-09-15T21:11:00.989916+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:00.989916+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9504, 'log_dups_size': 0, 'ondisk_log_size': 9504, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24941836, 'num_objects': 3045, 'num_object_clones': 0, 'num_object_copies': 12180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3045, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9502, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 730}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:03.935 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:04.700 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:04.700 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32235/196944 objects degraded (16.368%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":530,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":49236,"data_bytes":403300898,"bytes_used":2786861056,"bytes_avail":383760195584,"bytes_total":386547056640,"degraded_objects":32235,"degraded_total":196944,"degraded_ratio":0.16367596880331464,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:04.727 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:04.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:05.334 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:05.335 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:05.364 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-15T21:11:04.095956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":403300898,"num_objects":49236,"num_object_clones":0,"num_object_copies":196944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135842,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151087,"ondisk_log_size":151087,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721544,"kb_used_data":791984,"kb_used_omap":14,"kb_used_meta":1929457,"kb_avail":374765816,"statfs":{"total":386547056640,"available":383760195584,"internally_reserved":0,"allocated":810991616,"data_stored":810299208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":1975764530},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,22,18,16,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20398080,"num_objects":-2490,"num_object_clones":0,"num_object_copies":-9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2490,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004849"},"pg_stats":[{"pgid":"1.f","version":"35'9287","reported_seq":9337,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.414555+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:02.414555+0000","last_peered":"2024-09-15T21:11:02.414555+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:02.414555+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9287,"log_dups_size":0,"ondisk_log_size":9287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25108480,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9287,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":789}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9503","reported_seq":14395,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:00.475524+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:00.475524+0000","last_peered":"2024-09-15T21:11:00.475524+0000","last_clean":"2024-09-15T21:11:00.475524+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:00.475524+0000","last_undegraded":"2024-09-15T21:11:00.475524+0000","last_fullsized":"2024-09-15T21:11:00.475524+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24961024,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9271","reported_seq":9356,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.984168+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:10:58.984168+0000","last_peered":"2024-09-15T21:10:58.984168+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:10:58.984168+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25893534,"num_objects":3162,"num_object_clones":0,"num_object_copies":12648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9270,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":904}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9449","reported_seq":9523,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.235201+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:01.235201+0000","last_peered":"2024-09-15T21:11:01.235201+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:01.235201+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9449,"log_dups_size":0,"ondisk_log_size":9449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24630540,"num_objects":3007,"num_object_clones":0,"num_object_copies":12028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9447,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":673}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9534","reported_seq":9595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.380491+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:02.380491+0000","last_peered":"2024-09-15T21:11:02.380491+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:02.380491+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9534,"log_dups_size":0,"ondisk_log_size":9534,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25597196,"num_objects":3125,"num_object_clones":0,"num_object_copies":12500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9532,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":814}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9372","reported_seq":9422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.250116+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:02.250116+0000","last_peered":"2024-09-15T21:11:02.250116+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:02.250116+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24608768,"num_objects":3004,"num_object_clones":0,"num_object_copies":12016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9372,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9626","reported_seq":9701,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.321220+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:01.321220+0000","last_peered":"2024-09-15T21:11:01.321220+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:01.321220+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25769228,"num_objects":3146,"num_object_clones":0,"num_object_copies":12584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":780}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9706","reported_seq":9780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:01.331468+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:01.331468+0000","last_peered":"2024-09-15T21:11:01.331468+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:01.331468+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25023756,"num_objects":3055,"num_object_clones":0,"num_object_copies":12220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":581}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9423","reported_seq":9488,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.164791+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:02.164791+0000","last_peered":"2024-09-15T21:11:02.164791+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:02.164791+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9423,"log_dups_size":0,"ondisk_log_size":9423,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24807960,"num_objects":3029,"num_object_clones":0,"num_object_copies":12116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9420,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":697}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9480","reported_seq":9566,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.088217+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:10:59.088217+0000","last_peered":"2024-09-15T21:10:59.088217+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:10:59.088217+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9480,"log_dups_size":0,"ondisk_log_size":9480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25244940,"num_objects":3082,"num_object_clones":0,"num_object_copies":12328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9478,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":747}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9295","reported_seq":13963,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:02.224127+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:02.224127+0000","last_peered":"2024-09-15T21:11:02.224127+0000","last_clean":"2024-09-15T21:11:02.224127+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:02.224127+0000","last_undegraded":"2024-09-15T21:11:02.224127+0000","last_fullsized":"2024-09-15T21:11:02.224127+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9295,"log_dups_size":0,"ondisk_log_size":9295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25108480,"num_objects":3065,"num_object_clones":0,"num_object_copies":12260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3406,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9340","reported_seq":9384,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.406928+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:02.406928+0000","last_peered":"2024-09-15T21:11:02.406928+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:02.406928+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9340,"log_dups_size":0,"ondisk_log_size":9340,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25287302,"num_objects":3087,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9339,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9384","reported_seq":9882,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:59.043937+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:10:59.043937+0000","last_peered":"2024-09-15T21:10:59.043937+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:10:59.043937+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9384,"log_dups_size":0,"ondisk_log_size":9384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25533062,"num_objects":3117,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2081,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9383,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":207,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1036}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9410","reported_seq":9472,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:02.082884+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:02.082884+0000","last_peered":"2024-09-15T21:11:02.082884+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:02.082884+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9410,"log_dups_size":0,"ondisk_log_size":9410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24753420,"num_objects":3022,"num_object_clones":0,"num_object_copies":12088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9408,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":743}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9503","reported_seq":9591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:10:58.845188+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:10:58.845188+0000","last_peered":"2024-09-15T21:10:58.845188+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:10:58.845188+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9503,"log_dups_size":0,"ondisk_log_size":9503,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26031372,"num_objects":3178,"num_object_clones":0,"num_object_copies":12712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9501,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":897}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9504","reported_seq":9578,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:00.989916+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:00.989916+0000","last_peered":"2024-09-15T21:11:00.989916+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:00.989916+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9504,"log_dups_size":0,"ondisk_log_size":9504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24941836,"num_objects":3045,"num_object_clones":0,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9502,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":730}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":403300898,"num_objects":49236,"num_object_clones":0,"num_object_copies":196944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49236,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135842,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":228,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":807776256,"data_stored":807776256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":151087,"ondisk_log_size":151087,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705456,"kb_used_data":196184,"kb_used_omap":4,"kb_used_meta":509243,"kb_avail":93666384,"statfs":{"total":96636764160,"available":95914377216,"internally_reserved":0,"allocated":200892416,"data_stored":200719314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":521465552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,14,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708076,"kb_used_data":199308,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93663764,"statfs":{"total":96636764160,"available":95911694336,"internally_reserved":0,"allocated":204091392,"data_stored":203918290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,5,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051669,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":198628,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":203395072,"data_stored":203221970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247907,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694584,"kb_used_data":197864,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93677256,"statfs":{"total":96636764160,"available":95925510144,"internally_reserved":0,"allocated":202612736,"data_stored":202439634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":508620494},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201461760,"data_stored":201461760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":202321920,"data_stored":202321920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":203440128,"data_stored":203440128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":200552448,"data_stored":200552448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:08.367 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:08.995 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:08.995 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:09.019 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-15T21:11:08.101091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394748450,"num_objects":48192,"num_object_clones":0,"num_object_copies":192768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136886,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152131,"ondisk_log_size":152131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754928,"kb_used_data":775384,"kb_used_omap":14,"kb_used_meta":1979441,"kb_avail":374732432,"statfs":{"total":386547056640,"available":383726010368,"internally_reserved":0,"allocated":793993216,"data_stored":793300808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2026948146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,17,15,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22970368,"num_objects":-2804,"num_object_clones":0,"num_object_copies":-11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008251"},"pg_stats":[{"pgid":"1.f","version":"35'9355","reported_seq":9406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.312115+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:07.312115+0000","last_peered":"2024-09-15T21:11:07.312115+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:07.312115+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9553","reported_seq":14446,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:05.599831+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:05.599831+0000","last_peered":"2024-09-15T21:11:05.599831+0000","last_clean":"2024-09-15T21:11:05.599831+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:05.599831+0000","last_undegraded":"2024-09-15T21:11:05.599831+0000","last_fullsized":"2024-09-15T21:11:05.599831+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9332","reported_seq":9416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:03.841611+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:03.841611+0000","last_peered":"2024-09-15T21:11:03.841611+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:03.841611+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9332,"log_dups_size":0,"ondisk_log_size":9332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25393822,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9331,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9516","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.223295+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:06.223295+0000","last_peered":"2024-09-15T21:11:06.223295+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:06.223295+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24081676,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9601","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.352164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:07.352164+0000","last_peered":"2024-09-15T21:11:07.352164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:07.352164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048332,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9599,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9432","reported_seq":9482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.403174+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:07.403174+0000","last_peered":"2024-09-15T21:11:07.403174+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:07.403174+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9686","reported_seq":9761,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.282687+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:06.282687+0000","last_peered":"2024-09-15T21:11:06.282687+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:06.282687+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277708,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9774","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.381496+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:06.381496+0000","last_peered":"2024-09-15T21:11:06.381496+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:06.381496+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466700,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9471","reported_seq":9536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.367038+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:07.367038+0000","last_peered":"2024-09-15T21:11:07.367038+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:07.367038+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24414744,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9560","reported_seq":9644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.103263+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:04.103263+0000","last_peered":"2024-09-15T21:11:04.103263+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:04.103263+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589580,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9366","reported_seq":14034,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:07.457941+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:07.457941+0000","last_peered":"2024-09-15T21:11:07.457941+0000","last_clean":"2024-09-15T21:11:07.457941+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:07.457941+0000","last_undegraded":"2024-09-15T21:11:07.457941+0000","last_fullsized":"2024-09-15T21:11:07.457941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9397","reported_seq":9442,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.433121+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:07.433121+0000","last_peered":"2024-09-15T21:11:07.433121+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:07.433121+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24820358,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9396,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":774}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10054,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.128598+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:04.128598+0000","last_peered":"2024-09-15T21:11:04.128598+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:04.128598+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24959622,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9476","reported_seq":9538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.237284+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:07.237284+0000","last_peered":"2024-09-15T21:11:07.237284+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:07.237284+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212748,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9573","reported_seq":9661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.116469+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:04.116469+0000","last_peered":"2024-09-15T21:11:04.116469+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:04.116469+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25457932,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9585","reported_seq":9659,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.377568+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:06.377568+0000","last_peered":"2024-09-15T21:11:06.377568+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:06.377568+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278284,"num_objects":2964,"num_object_clones":0,"num_object_copies":11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":394748450,"num_objects":48192,"num_object_clones":0,"num_object_copies":192768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136886,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":790900736,"data_stored":790900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152131,"ondisk_log_size":152131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737480,"kb_used_data":192120,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93634360,"statfs":{"total":96636764160,"available":95881584640,"internally_reserved":0,"allocated":196730880,"data_stored":196557778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709448,"kb_used_data":195128,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93662392,"statfs":{"total":96636764160,"available":95910289408,"internally_reserved":0,"allocated":199811072,"data_stored":199637970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,0,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":194456,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":199122944,"data_stored":198949842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,7,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694572,"kb_used_data":193680,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93677268,"statfs":{"total":96636764160,"available":95925522432,"internally_reserved":0,"allocated":198328320,"data_stored":198155218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197296128,"data_stored":197296128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198082560,"data_stored":198082560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199159808,"data_stored":199159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196362240,"data_stored":196362240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:09.020 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:09.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:09.598 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:09.617 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-15T21:11:08.101091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":394748450,"num_objects":48192,"num_object_clones":0,"num_object_copies":192768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136886,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152131,"ondisk_log_size":152131,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754928,"kb_used_data":775384,"kb_used_omap":14,"kb_used_meta":1979441,"kb_avail":374732432,"statfs":{"total":386547056640,"available":383726010368,"internally_reserved":0,"allocated":793993216,"data_stored":793300808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2026948146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,17,15,14,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22970368,"num_objects":-2804,"num_object_clones":0,"num_object_copies":-11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2804,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008251"},"pg_stats":[{"pgid":"1.f","version":"35'9355","reported_seq":9406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.312115+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:07.312115+0000","last_peered":"2024-09-15T21:11:07.312115+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:07.312115+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9553","reported_seq":14446,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:05.599831+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:05.599831+0000","last_peered":"2024-09-15T21:11:05.599831+0000","last_clean":"2024-09-15T21:11:05.599831+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:05.599831+0000","last_undegraded":"2024-09-15T21:11:05.599831+0000","last_fullsized":"2024-09-15T21:11:05.599831+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9332","reported_seq":9416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:03.841611+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:03.841611+0000","last_peered":"2024-09-15T21:11:03.841611+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:03.841611+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9332,"log_dups_size":0,"ondisk_log_size":9332,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25393822,"num_objects":3101,"num_object_clones":0,"num_object_copies":12404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9331,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":843}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9516","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.223295+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:06.223295+0000","last_peered":"2024-09-15T21:11:06.223295+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:06.223295+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24081676,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9601","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.352164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:07.352164+0000","last_peered":"2024-09-15T21:11:07.352164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:07.352164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048332,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9599,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9432","reported_seq":9482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.403174+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:07.403174+0000","last_peered":"2024-09-15T21:11:07.403174+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:07.403174+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9686","reported_seq":9761,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.282687+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:06.282687+0000","last_peered":"2024-09-15T21:11:06.282687+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:06.282687+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277708,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9774","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.381496+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:06.381496+0000","last_peered":"2024-09-15T21:11:06.381496+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:06.381496+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466700,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9471","reported_seq":9536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.367038+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:07.367038+0000","last_peered":"2024-09-15T21:11:07.367038+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:07.367038+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24414744,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9560","reported_seq":9644,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.103263+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:04.103263+0000","last_peered":"2024-09-15T21:11:04.103263+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:04.103263+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9560,"log_dups_size":0,"ondisk_log_size":9560,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24589580,"num_objects":3002,"num_object_clones":0,"num_object_copies":12008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":667}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9366","reported_seq":14034,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:07.457941+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:07.457941+0000","last_peered":"2024-09-15T21:11:07.457941+0000","last_clean":"2024-09-15T21:11:07.457941+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:07.457941+0000","last_undegraded":"2024-09-15T21:11:07.457941+0000","last_fullsized":"2024-09-15T21:11:07.457941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9397","reported_seq":9442,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.433121+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:07.433121+0000","last_peered":"2024-09-15T21:11:07.433121+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:07.433121+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24820358,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9396,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":774}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9454","reported_seq":10054,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.128598+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:04.128598+0000","last_peered":"2024-09-15T21:11:04.128598+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:04.128598+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9454,"log_dups_size":0,"ondisk_log_size":9454,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24959622,"num_objects":3047,"num_object_clones":0,"num_object_copies":12188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9453,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9476","reported_seq":9538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.237284+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:07.237284+0000","last_peered":"2024-09-15T21:11:07.237284+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:07.237284+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212748,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9573","reported_seq":9661,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:04.116469+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:04.116469+0000","last_peered":"2024-09-15T21:11:04.116469+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:04.116469+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9573,"log_dups_size":0,"ondisk_log_size":9573,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25457932,"num_objects":3108,"num_object_clones":0,"num_object_copies":12432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9571,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9585","reported_seq":9659,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.377568+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:06.377568+0000","last_peered":"2024-09-15T21:11:06.377568+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:06.377568+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278284,"num_objects":2964,"num_object_clones":0,"num_object_copies":11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":394748450,"num_objects":48192,"num_object_clones":0,"num_object_copies":192768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136886,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":279,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":790900736,"data_stored":790900736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152131,"ondisk_log_size":152131,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737480,"kb_used_data":192120,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93634360,"statfs":{"total":96636764160,"available":95881584640,"internally_reserved":0,"allocated":196730880,"data_stored":196557778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709448,"kb_used_data":195128,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":93662392,"statfs":{"total":96636764160,"available":95910289408,"internally_reserved":0,"allocated":199811072,"data_stored":199637970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,8,0,5,6,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":194456,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":199122944,"data_stored":198949842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,7,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694572,"kb_used_data":193680,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93677268,"statfs":{"total":96636764160,"available":95925522432,"internally_reserved":0,"allocated":198328320,"data_stored":198155218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197296128,"data_stored":197296128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198082560,"data_stored":198082560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":199159808,"data_stored":199159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196362240,"data_stored":196362240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:09.618 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:09.618 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9355", 'reported_seq': 9406, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.312115+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:07.312115+0000', 'last_peered': '2024-09-15T21:11:07.312115+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:07.312115+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9355, 'log_dups_size': 0, 'ondisk_log_size': 9355, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24551424, 'num_objects': 2997, 'num_object_clones': 0, 'num_object_copies': 11988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2997, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9355, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 721}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9332", 'reported_seq': 9416, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:03.841611+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:03.841611+0000', 'last_peered': '2024-09-15T21:11:03.841611+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:03.841611+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9332, 'log_dups_size': 0, 'ondisk_log_size': 9332, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25393822, 'num_objects': 3101, 'num_object_clones': 0, 'num_object_copies': 12404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3101, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9331, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 843}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9516", 'reported_seq': 9588, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:06.223295+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:06.223295+0000', 'last_peered': '2024-09-15T21:11:06.223295+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:06.223295+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9516, 'log_dups_size': 0, 'ondisk_log_size': 9516, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24081676, 'num_objects': 2940, 'num_object_clones': 0, 'num_object_copies': 11760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2940, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9514, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 606}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9601", 'reported_seq': 9662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.352164+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:07.352164+0000', 'last_peered': '2024-09-15T21:11:07.352164+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:07.352164+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9601, 'log_dups_size': 0, 'ondisk_log_size': 9601, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25048332, 'num_objects': 3058, 'num_object_clones': 0, 'num_object_copies': 12232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9599, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 747}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9432", 'reported_seq': 9482, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.403174+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:07.403174+0000', 'last_peered': '2024-09-15T21:11:07.403174+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:07.403174+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9432, 'log_dups_size': 0, 'ondisk_log_size': 9432, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24117248, 'num_objects': 2944, 'num_object_clones': 0, 'num_object_copies': 11776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2944, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9432, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 628}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9686", 'reported_seq': 9761, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:06.282687+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:06.282687+0000', 'last_peered': '2024-09-15T21:11:06.282687+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:06.282687+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9686, 'log_dups_size': 0, 'ondisk_log_size': 9686, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25277708, 'num_objects': 3086, 'num_object_clones': 0, 'num_object_copies': 12344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9684, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 720}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:09.619 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9774", 'reported_seq': 9848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:06.381496+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:06.381496+0000', 'last_peered': '2024-09-15T21:11:06.381496+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:06.381496+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9774, 'log_dups_size': 0, 'ondisk_log_size': 9774, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24466700, 'num_objects': 2987, 'num_object_clones': 0, 'num_object_copies': 11948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9772, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9471", 'reported_seq': 9536, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.367038+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:07.367038+0000', 'last_peered': '2024-09-15T21:11:07.367038+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:07.367038+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9471, 'log_dups_size': 0, 'ondisk_log_size': 9471, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24414744, 'num_objects': 2981, 'num_object_clones': 0, 'num_object_copies': 11924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2981, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9468, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 649}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9560", 'reported_seq': 9644, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:04.103263+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:04.103263+0000', 'last_peered': '2024-09-15T21:11:04.103263+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:04.103263+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9560, 'log_dups_size': 0, 'ondisk_log_size': 9560, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24589580, 'num_objects': 3002, 'num_object_clones': 0, 'num_object_copies': 12008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 667}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9397", 'reported_seq': 9442, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.433121+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:07.433121+0000', 'last_peered': '2024-09-15T21:11:07.433121+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:07.433121+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9397, 'log_dups_size': 0, 'ondisk_log_size': 9397, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24820358, 'num_objects': 3030, 'num_object_clones': 0, 'num_object_copies': 12120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3030, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9396, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 774}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9454", 'reported_seq': 10054, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:04.128598+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:04.128598+0000', 'last_peered': '2024-09-15T21:11:04.128598+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:04.128598+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9454, 'log_dups_size': 0, 'ondisk_log_size': 9454, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24959622, 'num_objects': 3047, 'num_object_clones': 0, 'num_object_copies': 12188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2030, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9453, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 258, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9476", 'reported_seq': 9538, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:07.237284+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:07.237284+0000', 'last_peered': '2024-09-15T21:11:07.237284+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:07.237284+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9476, 'log_dups_size': 0, 'ondisk_log_size': 9476, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24212748, 'num_objects': 2956, 'num_object_clones': 0, 'num_object_copies': 11824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9474, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 677}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:09.620 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9573", 'reported_seq': 9661, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:04.116469+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:04.116469+0000', 'last_peered': '2024-09-15T21:11:04.116469+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:04.116469+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9573, 'log_dups_size': 0, 'ondisk_log_size': 9573, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 25457932, 'num_objects': 3108, 'num_object_clones': 0, 'num_object_copies': 12432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3108, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9571, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:09.621 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9585", 'reported_seq': 9659, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:06.377568+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:06.377568+0000', 'last_peered': '2024-09-15T21:11:06.377568+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:06.377568+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9585, 'log_dups_size': 0, 'ondisk_log_size': 9585, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24278284, 'num_objects': 2964, 'num_object_clones': 0, 'num_object_copies': 11856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2964, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9583, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:09.621 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:10.441 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:10.441 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32136/191788 objects degraded (16.756%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":536,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47947,"data_bytes":392741410,"bytes_used":2839478272,"bytes_avail":383707578368,"bytes_total":386547056640,"degraded_objects":32136,"degraded_total":191788,"degraded_ratio":0.16756001418232633,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:10.473 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:10.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:11.103 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:11.104 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:11.131 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-15T21:11:10.104088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392741410,"num_objects":47947,"num_object_clones":0,"num_object_copies":191788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137131,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152376,"ondisk_log_size":152376,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772928,"kb_used_data":771168,"kb_used_omap":14,"kb_used_meta":2001649,"kb_avail":374714432,"statfs":{"total":386547056640,"available":383707578368,"internally_reserved":0,"allocated":789676032,"data_stored":788983624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2049689138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,6,4,13,24,14,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18825216,"num_objects":-2298,"num_object_clones":0,"num_object_copies":-9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-152,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2298,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010493"},"pg_stats":[{"pgid":"1.f","version":"35'9355","reported_seq":9406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.312115+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:07.312115+0000","last_peered":"2024-09-15T21:11:07.312115+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:07.312115+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9355,"log_dups_size":0,"ondisk_log_size":9355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9355,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9553","reported_seq":14446,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:05.599831+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:05.599831+0000","last_peered":"2024-09-15T21:11:05.599831+0000","last_clean":"2024-09-15T21:11:05.599831+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:05.599831+0000","last_undegraded":"2024-09-15T21:11:05.599831+0000","last_fullsized":"2024-09-15T21:11:05.599831+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9553,"log_dups_size":0,"ondisk_log_size":9553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24551424,"num_objects":2997,"num_object_clones":0,"num_object_copies":11988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9401","reported_seq":9486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.131773+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:09.131773+0000","last_peered":"2024-09-15T21:11:09.131773+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:09.131773+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9401,"log_dups_size":0,"ondisk_log_size":9401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24828574,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9400,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9516","reported_seq":9588,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.223295+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:06.223295+0000","last_peered":"2024-09-15T21:11:06.223295+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:06.223295+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9516,"log_dups_size":0,"ondisk_log_size":9516,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24081676,"num_objects":2940,"num_object_clones":0,"num_object_copies":11760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9514,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":606}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9601","reported_seq":9662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.352164+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:07.352164+0000","last_peered":"2024-09-15T21:11:07.352164+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:07.352164+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9601,"log_dups_size":0,"ondisk_log_size":9601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25048332,"num_objects":3058,"num_object_clones":0,"num_object_copies":12232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9599,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":747}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9432","reported_seq":9482,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.403174+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:07.403174+0000","last_peered":"2024-09-15T21:11:07.403174+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:07.403174+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9432,"log_dups_size":0,"ondisk_log_size":9432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":2944,"num_object_clones":0,"num_object_copies":11776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9432,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":628}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9686","reported_seq":9761,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.282687+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:06.282687+0000","last_peered":"2024-09-15T21:11:06.282687+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:06.282687+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9686,"log_dups_size":0,"ondisk_log_size":9686,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25277708,"num_objects":3086,"num_object_clones":0,"num_object_copies":12344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9774","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.381496+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:06.381496+0000","last_peered":"2024-09-15T21:11:06.381496+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:06.381496+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24466700,"num_objects":2987,"num_object_clones":0,"num_object_copies":11948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9772,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9471","reported_seq":9536,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.367038+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:07.367038+0000","last_peered":"2024-09-15T21:11:07.367038+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:07.367038+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9471,"log_dups_size":0,"ondisk_log_size":9471,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24414744,"num_objects":2981,"num_object_clones":0,"num_object_copies":11924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9626","reported_seq":9713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.187460+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:09.187460+0000","last_peered":"2024-09-15T21:11:09.187460+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:09.187460+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048908,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":601}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9366","reported_seq":14034,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:07.457941+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:07.457941+0000","last_peered":"2024-09-15T21:11:07.457941+0000","last_clean":"2024-09-15T21:11:07.457941+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:07.457941+0000","last_undegraded":"2024-09-15T21:11:07.457941+0000","last_fullsized":"2024-09-15T21:11:07.457941+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9366,"log_dups_size":0,"ondisk_log_size":9366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24526848,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3477,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9397","reported_seq":9442,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.433121+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:07.433121+0000","last_peered":"2024-09-15T21:11:07.433121+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:07.433121+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9397,"log_dups_size":0,"ondisk_log_size":9397,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24820358,"num_objects":3030,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9396,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":774}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9507","reported_seq":10203,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.184013+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:09.184013+0000","last_peered":"2024-09-15T21:11:09.184013+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:09.184013+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24525446,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9476","reported_seq":9538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:07.237284+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:07.237284+0000","last_peered":"2024-09-15T21:11:07.237284+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:07.237284+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9476,"log_dups_size":0,"ondisk_log_size":9476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24212748,"num_objects":2956,"num_object_clones":0,"num_object_copies":11824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":677}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9630","reported_seq":9718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.172529+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:09.172529+0000","last_peered":"2024-09-15T21:11:09.172529+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:09.172529+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24990988,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9585","reported_seq":9659,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:06.377568+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:06.377568+0000","last_peered":"2024-09-15T21:11:06.377568+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:06.377568+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9585,"log_dups_size":0,"ondisk_log_size":9585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24278284,"num_objects":2964,"num_object_clones":0,"num_object_copies":11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9583,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":392741410,"num_objects":47947,"num_object_clones":0,"num_object_copies":191788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47947,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137131,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":786595840,"data_stored":786595840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152376,"ondisk_log_size":152376,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737480,"kb_used_data":192120,"kb_used_omap":4,"kb_used_meta":545339,"kb_avail":93634360,"statfs":{"total":96636764160,"available":95881584640,"internally_reserved":0,"allocated":196730880,"data_stored":196557778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":558427856},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,6,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727448,"kb_used_data":190912,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93644392,"statfs":{"total":96636764160,"available":95891857408,"internally_reserved":0,"allocated":195493888,"data_stored":195320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,9,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051670,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613428,"kb_used_data":194456,"kb_used_omap":4,"kb_used_meta":418939,"kb_avail":93758412,"statfs":{"total":96636764160,"available":96008613888,"internally_reserved":0,"allocated":199122944,"data_stored":198949842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":428994255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,7,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247908,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694572,"kb_used_data":193680,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93677268,"statfs":{"total":96636764160,"available":95925522432,"internally_reserved":0,"allocated":198328320,"data_stored":198155218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":512880334},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":197296128,"data_stored":197296128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":198082560,"data_stored":198082560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194854912,"data_stored":194854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":196362240,"data_stored":196362240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:14.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:14.723 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:14.723 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:14.757 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-15T21:11:14.106601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386408994,"num_objects":47174,"num_object_clones":0,"num_object_copies":188696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137904,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153149,"ondisk_log_size":153149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761588,"kb_used_data":758428,"kb_used_omap":14,"kb_used_meta":2003057,"kb_avail":374725772,"statfs":{"total":386547056640,"available":383719190528,"internally_reserved":0,"allocated":776630272,"data_stored":775937864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2051130930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,21,28,20,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20619264,"num_objects":-2517,"num_object_clones":0,"num_object_copies":-10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011347"},"pg_stats":[{"pgid":"1.f","version":"35'9428","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.503326+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:12.503326+0000","last_peered":"2024-09-15T21:11:12.503326+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:12.503326+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23953408,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9609","reported_seq":14502,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:10.426400+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:10.426400+0000","last_peered":"2024-09-15T21:11:10.426400+0000","last_clean":"2024-09-15T21:11:10.426400+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:10.426400+0000","last_undegraded":"2024-09-15T21:11:10.426400+0000","last_fullsized":"2024-09-15T21:11:10.426400+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24092672,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9401","reported_seq":9486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.131773+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:09.131773+0000","last_peered":"2024-09-15T21:11:09.131773+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:09.131773+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9401,"log_dups_size":0,"ondisk_log_size":9401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24828574,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9400,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9578","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.473175+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:11.473175+0000","last_peered":"2024-09-15T21:11:11.473175+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:11.473175+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573772,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9674","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.440377+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:12.440377+0000","last_peered":"2024-09-15T21:11:12.440377+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:12.440377+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9674,"log_dups_size":0,"ondisk_log_size":9674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24450316,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9487","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.435230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:12.435230+0000","last_peered":"2024-09-15T21:11:12.435230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:12.435230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":2889,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9747","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.143614+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:11.143614+0000","last_peered":"2024-09-15T21:11:11.143614+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:11.143614+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9850","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.300352+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:11.300352+0000","last_peered":"2024-09-15T21:11:11.300352+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:11.300352+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23844108,"num_objects":2911,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9537","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.511383+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:12.511383+0000","last_peered":"2024-09-15T21:11:12.511383+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:12.511383+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23874072,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9626","reported_seq":9713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.187460+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:09.187460+0000","last_peered":"2024-09-15T21:11:09.187460+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:09.187460+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048908,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":601}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9427","reported_seq":14095,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:12.420034+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:12.420034+0000","last_peered":"2024-09-15T21:11:12.420034+0000","last_clean":"2024-09-15T21:11:12.420034+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:12.420034+0000","last_undegraded":"2024-09-15T21:11:12.420034+0000","last_fullsized":"2024-09-15T21:11:12.420034+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9427,"log_dups_size":0,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24027136,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9459","reported_seq":9504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.501893+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:12.501893+0000","last_peered":"2024-09-15T21:11:12.501893+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:12.501893+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9459,"log_dups_size":0,"ondisk_log_size":9459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24312454,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9507","reported_seq":10203,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.184013+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:09.184013+0000","last_peered":"2024-09-15T21:11:09.184013+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:09.184013+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24525446,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9540","reported_seq":9600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.473358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:12.473358+0000","last_peered":"2024-09-15T21:11:12.473358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:12.473358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9540,"log_dups_size":0,"ondisk_log_size":9540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9630","reported_seq":9718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.172529+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:09.172529+0000","last_peered":"2024-09-15T21:11:09.172529+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:09.172529+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24990988,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9649","reported_seq":9723,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.459733+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:11.459733+0000","last_peered":"2024-09-15T21:11:11.459733+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:11.459733+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":386408994,"num_objects":47174,"num_object_clones":0,"num_object_copies":188696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137904,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774021120,"data_stored":774021120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153149,"ondisk_log_size":153149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":188236,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":192753664,"data_stored":192580562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":568389328},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727448,"kb_used_data":190912,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93644392,"statfs":{"total":96636764160,"available":95891857408,"internally_reserved":0,"allocated":195493888,"data_stored":195320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,9,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596192,"kb_used_data":189704,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93775648,"statfs":{"total":96636764160,"available":96026263552,"internally_reserved":0,"allocated":194256896,"data_stored":194083794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694624,"kb_used_data":189576,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93677216,"statfs":{"total":96636764160,"available":95925469184,"internally_reserved":0,"allocated":194125824,"data_stored":193952722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193126400,"data_stored":193126400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193859584,"data_stored":193859584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194854912,"data_stored":194854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192180224,"data_stored":192180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:14.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:15.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:15.384 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:15.407 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-15T21:11:14.106601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":386408994,"num_objects":47174,"num_object_clones":0,"num_object_copies":188696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137904,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153149,"ondisk_log_size":153149,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761588,"kb_used_data":758428,"kb_used_omap":14,"kb_used_meta":2003057,"kb_avail":374725772,"statfs":{"total":386547056640,"available":383719190528,"internally_reserved":0,"allocated":776630272,"data_stored":775937864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2051130930},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,6,21,28,20,14,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20619264,"num_objects":-2517,"num_object_clones":0,"num_object_copies":-10068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011347"},"pg_stats":[{"pgid":"1.f","version":"35'9428","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.503326+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:12.503326+0000","last_peered":"2024-09-15T21:11:12.503326+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:12.503326+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23953408,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9609","reported_seq":14502,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:10.426400+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:10.426400+0000","last_peered":"2024-09-15T21:11:10.426400+0000","last_clean":"2024-09-15T21:11:10.426400+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:10.426400+0000","last_undegraded":"2024-09-15T21:11:10.426400+0000","last_fullsized":"2024-09-15T21:11:10.426400+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9609,"log_dups_size":0,"ondisk_log_size":9609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24092672,"num_objects":2941,"num_object_clones":0,"num_object_copies":11764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":275,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9401","reported_seq":9486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.131773+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:09.131773+0000","last_peered":"2024-09-15T21:11:09.131773+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:09.131773+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9401,"log_dups_size":0,"ondisk_log_size":9401,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24828574,"num_objects":3032,"num_object_clones":0,"num_object_copies":12128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9400,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":774}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9578","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.473175+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:11.473175+0000","last_peered":"2024-09-15T21:11:11.473175+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:11.473175+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573772,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9674","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.440377+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:12.440377+0000","last_peered":"2024-09-15T21:11:12.440377+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:12.440377+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9674,"log_dups_size":0,"ondisk_log_size":9674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24450316,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9487","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.435230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:12.435230+0000","last_peered":"2024-09-15T21:11:12.435230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:12.435230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":2889,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9747","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.143614+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:11.143614+0000","last_peered":"2024-09-15T21:11:11.143614+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:11.143614+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9850","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.300352+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:11.300352+0000","last_peered":"2024-09-15T21:11:11.300352+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:11.300352+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23844108,"num_objects":2911,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9537","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.511383+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:12.511383+0000","last_peered":"2024-09-15T21:11:12.511383+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:12.511383+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23874072,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9626","reported_seq":9713,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.187460+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:09.187460+0000","last_peered":"2024-09-15T21:11:09.187460+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:09.187460+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9626,"log_dups_size":0,"ondisk_log_size":9626,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24048908,"num_objects":2936,"num_object_clones":0,"num_object_copies":11744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9624,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":601}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9427","reported_seq":14095,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:12.420034+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:12.420034+0000","last_peered":"2024-09-15T21:11:12.420034+0000","last_clean":"2024-09-15T21:11:12.420034+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:12.420034+0000","last_undegraded":"2024-09-15T21:11:12.420034+0000","last_fullsized":"2024-09-15T21:11:12.420034+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9427,"log_dups_size":0,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24027136,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9459","reported_seq":9504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.501893+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:12.501893+0000","last_peered":"2024-09-15T21:11:12.501893+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:12.501893+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9459,"log_dups_size":0,"ondisk_log_size":9459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24312454,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9507","reported_seq":10203,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.184013+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:09.184013+0000","last_peered":"2024-09-15T21:11:09.184013+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:09.184013+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9507,"log_dups_size":0,"ondisk_log_size":9507,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24525446,"num_objects":2994,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1012}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9540","reported_seq":9600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.473358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:12.473358+0000","last_peered":"2024-09-15T21:11:12.473358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:12.473358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9540,"log_dups_size":0,"ondisk_log_size":9540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9630","reported_seq":9718,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:09.172529+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:09.172529+0000","last_peered":"2024-09-15T21:11:09.172529+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:09.172529+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9630,"log_dups_size":0,"ondisk_log_size":9630,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24990988,"num_objects":3051,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9628,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":770}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9649","reported_seq":9723,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.459733+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:11.459733+0000","last_peered":"2024-09-15T21:11:11.459733+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:11.459733+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":386408994,"num_objects":47174,"num_object_clones":0,"num_object_copies":188696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":137904,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":327,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774021120,"data_stored":774021120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153149,"ondisk_log_size":153149,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":188236,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":192753664,"data_stored":192580562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":568389328},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727448,"kb_used_data":190912,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93644392,"statfs":{"total":96636764160,"available":95891857408,"internally_reserved":0,"allocated":195493888,"data_stored":195320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,4,9,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051671,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596192,"kb_used_data":189704,"kb_used_omap":4,"kb_used_meta":406459,"kb_avail":93775648,"statfs":{"total":96636764160,"available":96026263552,"internally_reserved":0,"allocated":194256896,"data_stored":194083794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":416214735},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694624,"kb_used_data":189576,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93677216,"statfs":{"total":96636764160,"available":95925469184,"internally_reserved":0,"allocated":194125824,"data_stored":193952722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193126400,"data_stored":193126400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193859584,"data_stored":193859584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":194854912,"data_stored":194854912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192180224,"data_stored":192180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9428", 'reported_seq': 9479, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.503326+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:12.503326+0000', 'last_peered': '2024-09-15T21:11:12.503326+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:12.503326+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9428, 'log_dups_size': 0, 'ondisk_log_size': 9428, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23953408, 'num_objects': 2924, 'num_object_clones': 0, 'num_object_copies': 11696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9428, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 648}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9401", 'reported_seq': 9486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:09.131773+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:09.131773+0000', 'last_peered': '2024-09-15T21:11:09.131773+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:09.131773+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9401, 'log_dups_size': 0, 'ondisk_log_size': 9401, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24828574, 'num_objects': 3032, 'num_object_clones': 0, 'num_object_copies': 12128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3032, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9400, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 774}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9578", 'reported_seq': 9652, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:11.473175+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:11.473175+0000', 'last_peered': '2024-09-15T21:11:11.473175+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:11.473175+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9578, 'log_dups_size': 0, 'ondisk_log_size': 9578, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23573772, 'num_objects': 2878, 'num_object_clones': 0, 'num_object_copies': 11512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2878, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9576, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 544}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:15.408 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9674", 'reported_seq': 9735, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.440377+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:12.440377+0000', 'last_peered': '2024-09-15T21:11:12.440377+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:12.440377+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9674, 'log_dups_size': 0, 'ondisk_log_size': 9674, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24450316, 'num_objects': 2985, 'num_object_clones': 0, 'num_object_copies': 11940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2985, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9672, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 674}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9487", 'reported_seq': 9537, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.435230+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:12.435230+0000', 'last_peered': '2024-09-15T21:11:12.435230+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:12.435230+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9487, 'log_dups_size': 0, 'ondisk_log_size': 9487, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23666688, 'num_objects': 2889, 'num_object_clones': 0, 'num_object_copies': 11556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2889, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9487, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 573}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9747", 'reported_seq': 9822, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:11.143614+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:11.143614+0000', 'last_peered': '2024-09-15T21:11:11.143614+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:11.143614+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9747, 'log_dups_size': 0, 'ondisk_log_size': 9747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24777996, 'num_objects': 3025, 'num_object_clones': 0, 'num_object_copies': 12100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9745, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 659}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9850", 'reported_seq': 9924, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:11.300352+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:11.300352+0000', 'last_peered': '2024-09-15T21:11:11.300352+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:11.300352+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23844108, 'num_objects': 2911, 'num_object_clones': 0, 'num_object_copies': 11644, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2911, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 437}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9537", 'reported_seq': 9598, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.511383+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:12.511383+0000', 'last_peered': '2024-09-15T21:11:12.511383+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:12.511383+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9537, 'log_dups_size': 0, 'ondisk_log_size': 9537, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23874072, 'num_objects': 2915, 'num_object_clones': 0, 'num_object_copies': 11660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2915, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9534, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 583}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:15.409 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9626", 'reported_seq': 9713, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:09.187460+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:09.187460+0000', 'last_peered': '2024-09-15T21:11:09.187460+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:09.187460+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9626, 'log_dups_size': 0, 'ondisk_log_size': 9626, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24048908, 'num_objects': 2936, 'num_object_clones': 0, 'num_object_copies': 11744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2936, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9624, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 601}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9459", 'reported_seq': 9504, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.501893+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:12.501893+0000', 'last_peered': '2024-09-15T21:11:12.501893+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:12.501893+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9459, 'log_dups_size': 0, 'ondisk_log_size': 9459, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24312454, 'num_objects': 2968, 'num_object_clones': 0, 'num_object_copies': 11872, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2968, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9458, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9507", 'reported_seq': 10203, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:09.184013+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:09.184013+0000', 'last_peered': '2024-09-15T21:11:09.184013+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:09.184013+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9507, 'log_dups_size': 0, 'ondisk_log_size': 9507, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24525446, 'num_objects': 2994, 'num_object_clones': 0, 'num_object_copies': 11976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1982, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9506, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 306, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1012}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9540", 'reported_seq': 9600, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:12.473358+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:12.473358+0000', 'last_peered': '2024-09-15T21:11:12.473358+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:12.473358+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9540, 'log_dups_size': 0, 'ondisk_log_size': 9540, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23688460, 'num_objects': 2892, 'num_object_clones': 0, 'num_object_copies': 11568, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2892, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9538, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 613}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9630", 'reported_seq': 9718, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:09.172529+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:09.172529+0000', 'last_peered': '2024-09-15T21:11:09.172529+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:09.172529+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9630, 'log_dups_size': 0, 'ondisk_log_size': 9630, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24990988, 'num_objects': 3051, 'num_object_clones': 0, 'num_object_copies': 12204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3051, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9628, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 770}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:15.410 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9649", 'reported_seq': 9723, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:11.459733+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:11.459733+0000', 'last_peered': '2024-09-15T21:11:11.459733+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:11.459733+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9649, 'log_dups_size': 0, 'ondisk_log_size': 9649, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23753996, 'num_objects': 2900, 'num_object_clones': 0, 'num_object_copies': 11600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9647, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 585}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:15.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:16.162 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:16.163 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32136/188696 objects degraded (17.031%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":541,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":47174,"data_bytes":386408994,"bytes_used":2827866112,"bytes_avail":383719190528,"bytes_total":386547056640,"degraded_objects":32136,"degraded_total":188696,"degraded_ratio":0.1703056768558952,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":209},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:16.191 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:16.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:16.766 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:16.767 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:16.795 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-09-15T21:11:16.107471+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":383828514,"num_objects":46859,"num_object_clones":0,"num_object_copies":187436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138219,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153464,"ondisk_log_size":153464,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779684,"kb_used_data":750164,"kb_used_omap":14,"kb_used_meta":2029425,"kb_avail":374707676,"statfs":{"total":386547056640,"available":383700660224,"internally_reserved":0,"allocated":768167936,"data_stored":767475528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2078131762},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,4,22,22,18,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19472384,"num_objects":-2377,"num_object_clones":0,"num_object_copies":-9508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2377,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011515"},"pg_stats":[{"pgid":"1.f","version":"35'9428","reported_seq":9479,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.503326+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:12.503326+0000","last_peered":"2024-09-15T21:11:12.503326+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:12.503326+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9428,"log_dups_size":0,"ondisk_log_size":9428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23953408,"num_objects":2924,"num_object_clones":0,"num_object_copies":11696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9682","reported_seq":14575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:15.744090+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:15.744090+0000","last_peered":"2024-09-15T21:11:15.744090+0000","last_clean":"2024-09-15T21:11:15.744090+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:15.744090+0000","last_undegraded":"2024-09-15T21:11:15.744090+0000","last_fullsized":"2024-09-15T21:11:15.744090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23494656,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9472","reported_seq":9557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:14.165497+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:14.165497+0000","last_peered":"2024-09-15T21:11:14.165497+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:14.165497+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9472,"log_dups_size":0,"ondisk_log_size":9472,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24246942,"num_objects":2961,"num_object_clones":0,"num_object_copies":11844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2961,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9471,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":703}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9578","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.473175+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:11.473175+0000","last_peered":"2024-09-15T21:11:11.473175+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:11.473175+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9578,"log_dups_size":0,"ondisk_log_size":9578,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23573772,"num_objects":2878,"num_object_clones":0,"num_object_copies":11512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9576,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":544}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9674","reported_seq":9735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.440377+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:12.440377+0000","last_peered":"2024-09-15T21:11:12.440377+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:12.440377+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9674,"log_dups_size":0,"ondisk_log_size":9674,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24450316,"num_objects":2985,"num_object_clones":0,"num_object_copies":11940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9672,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9487","reported_seq":9537,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.435230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:12.435230+0000","last_peered":"2024-09-15T21:11:12.435230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:12.435230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9487,"log_dups_size":0,"ondisk_log_size":9487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23666688,"num_objects":2889,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9487,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":573}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9747","reported_seq":9822,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.143614+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:11.143614+0000","last_peered":"2024-09-15T21:11:11.143614+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:11.143614+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24777996,"num_objects":3025,"num_object_clones":0,"num_object_copies":12100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":659}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9850","reported_seq":9924,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.300352+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:11.300352+0000","last_peered":"2024-09-15T21:11:11.300352+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:11.300352+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23844108,"num_objects":2911,"num_object_clones":0,"num_object_copies":11644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":437}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9537","reported_seq":9598,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.511383+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:12.511383+0000","last_peered":"2024-09-15T21:11:12.511383+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:12.511383+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23874072,"num_objects":2915,"num_object_clones":0,"num_object_copies":11660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9534,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":583}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9683","reported_seq":9770,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:14.243430+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:14.243430+0000","last_peered":"2024-09-15T21:11:14.243430+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:14.243430+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9681,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":544}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9427","reported_seq":14095,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:12.420034+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:12.420034+0000","last_peered":"2024-09-15T21:11:12.420034+0000","last_clean":"2024-09-15T21:11:12.420034+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:12.420034+0000","last_undegraded":"2024-09-15T21:11:12.420034+0000","last_fullsized":"2024-09-15T21:11:12.420034+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9427,"log_dups_size":0,"ondisk_log_size":9427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24027136,"num_objects":2933,"num_object_clones":0,"num_object_copies":11732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9459","reported_seq":9504,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.501893+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:12.501893+0000","last_peered":"2024-09-15T21:11:12.501893+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:12.501893+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9459,"log_dups_size":0,"ondisk_log_size":9459,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24312454,"num_objects":2968,"num_object_clones":0,"num_object_copies":11872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9458,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":712}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9562","reported_seq":10385,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:14.371604+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:14.371604+0000","last_peered":"2024-09-15T21:11:14.371604+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:14.371604+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9562,"log_dups_size":0,"ondisk_log_size":9562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24074886,"num_objects":2939,"num_object_clones":0,"num_object_copies":11756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1918,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9561,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":370,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1021}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9540","reported_seq":9600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:12.473358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:12.473358+0000","last_peered":"2024-09-15T21:11:12.473358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:12.473358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9540,"log_dups_size":0,"ondisk_log_size":9540,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23688460,"num_objects":2892,"num_object_clones":0,"num_object_copies":11568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":613}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9689","reported_seq":9777,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:14.124353+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:14.124353+0000","last_peered":"2024-09-15T21:11:14.124353+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:14.124353+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9689,"log_dups_size":0,"ondisk_log_size":9689,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24507660,"num_objects":2992,"num_object_clones":0,"num_object_copies":11968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9687,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":711}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9649","reported_seq":9723,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:11.459733+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:11.459733+0000","last_peered":"2024-09-15T21:11:11.459733+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:11.459733+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9649,"log_dups_size":0,"ondisk_log_size":9649,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9647,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":585}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":383828514,"num_objects":46859,"num_object_clones":0,"num_object_copies":187436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32072,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46859,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":138219,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":765595648,"data_stored":765595648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153464,"ondisk_log_size":153464,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743324,"kb_used_data":188236,"kb_used_omap":4,"kb_used_meta":555067,"kb_avail":93628516,"statfs":{"total":96636764160,"available":95875600384,"internally_reserved":0,"allocated":192753664,"data_stored":192580562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":568389328},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745544,"kb_used_data":186800,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":93626296,"statfs":{"total":96636764160,"available":95873327104,"internally_reserved":0,"allocated":191283200,"data_stored":191110098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596192,"kb_used_data":185552,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93775648,"statfs":{"total":96636764160,"available":96026263552,"internally_reserved":0,"allocated":190005248,"data_stored":189832146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247909,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694624,"kb_used_data":189576,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93677216,"statfs":{"total":96636764160,"available":95925469184,"internally_reserved":0,"allocated":194125824,"data_stored":193952722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":517140174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2,8,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":193126400,"data_stored":193126400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189616128,"data_stored":189616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":190672896,"data_stored":190672896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":192180224,"data_stored":192180224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:19.797 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:20.379 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:20.380 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:20.409 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-15T21:11:20.108845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376144418,"num_objects":45921,"num_object_clones":0,"num_object_copies":183684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139157,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154402,"ondisk_log_size":154402,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798196,"kb_used_data":738128,"kb_used_omap":14,"kb_used_meta":2059953,"kb_avail":374689164,"statfs":{"total":386547056640,"available":383681703936,"internally_reserved":0,"allocated":755843072,"data_stored":755150664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2109392434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,4,22,23,18,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18604032,"num_objects":-2271,"num_object_clones":0,"num_object_copies":-9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2271,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007753"},"pg_stats":[{"pgid":"1.f","version":"35'9489","reported_seq":9540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.597895+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:17.597895+0000","last_peered":"2024-09-15T21:11:17.597895+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:17.597895+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23453696,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9489,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9682","reported_seq":14575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:15.744090+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:15.744090+0000","last_peered":"2024-09-15T21:11:15.744090+0000","last_clean":"2024-09-15T21:11:15.744090+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:15.744090+0000","last_undegraded":"2024-09-15T21:11:15.744090+0000","last_fullsized":"2024-09-15T21:11:15.744090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23494656,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9542","reported_seq":9627,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.245859+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:19.245859+0000","last_peered":"2024-09-15T21:11:19.245859+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:19.245859+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23673502,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9541,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9644","reported_seq":9719,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.523207+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:16.523207+0000","last_peered":"2024-09-15T21:11:16.523207+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:16.523207+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23033100,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9747","reported_seq":9807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.558361+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:17.558361+0000","last_peered":"2024-09-15T21:11:17.558361+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:17.558361+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852300,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9537","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.643620+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:17.643620+0000","last_peered":"2024-09-15T21:11:17.643620+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:17.643620+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9803","reported_seq":9878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.132834+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:16.132834+0000","last_peered":"2024-09-15T21:11:16.132834+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:16.132834+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9803,"log_dups_size":0,"ondisk_log_size":9803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24319244,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9801,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9909","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.336647+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:16.336647+0000","last_peered":"2024-09-15T21:11:16.336647+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:16.336647+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360780,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9603","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.275373+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:17.275373+0000","last_peered":"2024-09-15T21:11:17.275373+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:17.275373+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23333400,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9732","reported_seq":9819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.187488+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:19.187488+0000","last_peered":"2024-09-15T21:11:19.187488+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:19.187488+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23180556,"num_objects":2830,"num_object_clones":0,"num_object_copies":11320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9500","reported_seq":14167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:17.645535+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:17.645535+0000","last_peered":"2024-09-15T21:11:17.645535+0000","last_clean":"2024-09-15T21:11:17.645535+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:17.645535+0000","last_undegraded":"2024-09-15T21:11:17.645535+0000","last_fullsized":"2024-09-15T21:11:17.645535+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23429120,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9530","reported_seq":9575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.354987+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:17.354987+0000","last_peered":"2024-09-15T21:11:17.354987+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:17.354987+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9530,"log_dups_size":0,"ondisk_log_size":9530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23730822,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9529,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9613","reported_seq":10525,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.376670+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:19.376670+0000","last_peered":"2024-09-15T21:11:19.376670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:19.376670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23657094,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9603","reported_seq":9665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.563489+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:17.563489+0000","last_peered":"2024-09-15T21:11:17.563489+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:17.563489+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172364,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9751","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.172834+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:19.172834+0000","last_peered":"2024-09-15T21:11:19.172834+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:19.172834+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23999756,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9717","reported_seq":9791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.401328+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:16.401328+0000","last_peered":"2024-09-15T21:11:16.401328+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:16.401328+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196940,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9715,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":517}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376144418,"num_objects":45921,"num_object_clones":0,"num_object_copies":183684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139157,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753061888,"data_stored":753061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154402,"ondisk_log_size":154402,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":184272,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":188694528,"data_stored":188521426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745732,"kb_used_data":182820,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93626108,"statfs":{"total":96636764160,"available":95873134592,"internally_reserved":0,"allocated":187207680,"data_stored":187034578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596192,"kb_used_data":185552,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93775648,"statfs":{"total":96636764160,"available":96026263552,"internally_reserved":0,"allocated":190005248,"data_stored":189832146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712740,"kb_used_data":185484,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93659100,"statfs":{"total":96636764160,"available":95906918400,"internally_reserved":0,"allocated":189935616,"data_stored":189762514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188960768,"data_stored":188960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189616128,"data_stored":189616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186499072,"data_stored":186499072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187985920,"data_stored":187985920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:20.411 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:21.042 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:21.047 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:21.082 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-09-15T21:11:20.108845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376144418,"num_objects":45921,"num_object_clones":0,"num_object_copies":183684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139157,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154402,"ondisk_log_size":154402,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2798196,"kb_used_data":738128,"kb_used_omap":14,"kb_used_meta":2059953,"kb_avail":374689164,"statfs":{"total":386547056640,"available":383681703936,"internally_reserved":0,"allocated":755843072,"data_stored":755150664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2109392434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,4,22,23,18,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18604032,"num_objects":-2271,"num_object_clones":0,"num_object_copies":-9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2271,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007753"},"pg_stats":[{"pgid":"1.f","version":"35'9489","reported_seq":9540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.597895+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:17.597895+0000","last_peered":"2024-09-15T21:11:17.597895+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:17.597895+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23453696,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9489,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9682","reported_seq":14575,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:15.744090+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:15.744090+0000","last_peered":"2024-09-15T21:11:15.744090+0000","last_clean":"2024-09-15T21:11:15.744090+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:15.744090+0000","last_undegraded":"2024-09-15T21:11:15.744090+0000","last_fullsized":"2024-09-15T21:11:15.744090+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9682,"log_dups_size":0,"ondisk_log_size":9682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23494656,"num_objects":2868,"num_object_clones":0,"num_object_copies":11472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":348,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9542","reported_seq":9627,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.245859+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:19.245859+0000","last_peered":"2024-09-15T21:11:19.245859+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:19.245859+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23673502,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9541,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9644","reported_seq":9719,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.523207+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:16.523207+0000","last_peered":"2024-09-15T21:11:16.523207+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:16.523207+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23033100,"num_objects":2812,"num_object_clones":0,"num_object_copies":11248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":478}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9747","reported_seq":9807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.558361+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:17.558361+0000","last_peered":"2024-09-15T21:11:17.558361+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:17.558361+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852300,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9537","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.643620+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:17.643620+0000","last_peered":"2024-09-15T21:11:17.643620+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:17.643620+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9803","reported_seq":9878,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.132834+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:16.132834+0000","last_peered":"2024-09-15T21:11:16.132834+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:16.132834+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9803,"log_dups_size":0,"ondisk_log_size":9803,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24319244,"num_objects":2969,"num_object_clones":0,"num_object_copies":11876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9801,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":603}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9909","reported_seq":9983,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.336647+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:16.336647+0000","last_peered":"2024-09-15T21:11:16.336647+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:16.336647+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9909,"log_dups_size":0,"ondisk_log_size":9909,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360780,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9603","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.275373+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:17.275373+0000","last_peered":"2024-09-15T21:11:17.275373+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:17.275373+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23333400,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9732","reported_seq":9819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.187488+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:19.187488+0000","last_peered":"2024-09-15T21:11:19.187488+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:19.187488+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23180556,"num_objects":2830,"num_object_clones":0,"num_object_copies":11320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9500","reported_seq":14167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:17.645535+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:17.645535+0000","last_peered":"2024-09-15T21:11:17.645535+0000","last_clean":"2024-09-15T21:11:17.645535+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:17.645535+0000","last_undegraded":"2024-09-15T21:11:17.645535+0000","last_fullsized":"2024-09-15T21:11:17.645535+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23429120,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9530","reported_seq":9575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.354987+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:17.354987+0000","last_peered":"2024-09-15T21:11:17.354987+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:17.354987+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9530,"log_dups_size":0,"ondisk_log_size":9530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23730822,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9529,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9613","reported_seq":10525,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.376670+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:19.376670+0000","last_peered":"2024-09-15T21:11:19.376670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:19.376670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23657094,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9603","reported_seq":9665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.563489+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:17.563489+0000","last_peered":"2024-09-15T21:11:17.563489+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:17.563489+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172364,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9751","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.172834+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:19.172834+0000","last_peered":"2024-09-15T21:11:19.172834+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:19.172834+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23999756,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9717","reported_seq":9791,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:16.401328+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:16.401328+0000","last_peered":"2024-09-15T21:11:16.401328+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:16.401328+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9717,"log_dups_size":0,"ondisk_log_size":9717,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23196940,"num_objects":2832,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9715,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":517}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":376144418,"num_objects":45921,"num_object_clones":0,"num_object_copies":183684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45921,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139157,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":753061888,"data_stored":753061888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154402,"ondisk_log_size":154402,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":184272,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":188694528,"data_stored":188521426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745732,"kb_used_data":182820,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93626108,"statfs":{"total":96636764160,"available":95873134592,"internally_reserved":0,"allocated":187207680,"data_stored":187034578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051672,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596192,"kb_used_data":185552,"kb_used_omap":4,"kb_used_meta":410619,"kb_avail":93775648,"statfs":{"total":96636764160,"available":96026263552,"internally_reserved":0,"allocated":190005248,"data_stored":189832146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":420474575},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247910,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712740,"kb_used_data":185484,"kb_used_omap":4,"kb_used_meta":527227,"kb_avail":93659100,"statfs":{"total":96636764160,"available":95906918400,"internally_reserved":0,"allocated":189935616,"data_stored":189762514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":539881166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":188960768,"data_stored":188960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189616128,"data_stored":189616128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186499072,"data_stored":186499072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187985920,"data_stored":187985920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:21.083 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:21.083 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9489", 'reported_seq': 9540, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.597895+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:17.597895+0000', 'last_peered': '2024-09-15T21:11:17.597895+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:17.597895+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9489, 'log_dups_size': 0, 'ondisk_log_size': 9489, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23453696, 'num_objects': 2863, 'num_object_clones': 0, 'num_object_copies': 11452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2863, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9489, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 587}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9542", 'reported_seq': 9627, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:19.245859+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:19.245859+0000', 'last_peered': '2024-09-15T21:11:19.245859+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:19.245859+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9542, 'log_dups_size': 0, 'ondisk_log_size': 9542, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23673502, 'num_objects': 2891, 'num_object_clones': 0, 'num_object_copies': 11564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2891, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9541, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 633}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9644", 'reported_seq': 9719, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:16.523207+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:16.523207+0000', 'last_peered': '2024-09-15T21:11:16.523207+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:16.523207+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9644, 'log_dups_size': 0, 'ondisk_log_size': 9644, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23033100, 'num_objects': 2812, 'num_object_clones': 0, 'num_object_copies': 11248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9642, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 478}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9747", 'reported_seq': 9807, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.558361+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:17.558361+0000', 'last_peered': '2024-09-15T21:11:17.558361+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:17.558361+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9747, 'log_dups_size': 0, 'ondisk_log_size': 9747, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23852300, 'num_objects': 2912, 'num_object_clones': 0, 'num_object_copies': 11648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2912, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9745, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 601}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9537", 'reported_seq': 9587, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.643620+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:17.643620+0000', 'last_peered': '2024-09-15T21:11:17.643620+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:17.643620+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9537, 'log_dups_size': 0, 'ondisk_log_size': 9537, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23257088, 'num_objects': 2839, 'num_object_clones': 0, 'num_object_copies': 11356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2839, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9537, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 523}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:21.084 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9803", 'reported_seq': 9878, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:16.132834+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:16.132834+0000', 'last_peered': '2024-09-15T21:11:16.132834+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:16.132834+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9803, 'log_dups_size': 0, 'ondisk_log_size': 9803, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 24319244, 'num_objects': 2969, 'num_object_clones': 0, 'num_object_copies': 11876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2969, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9801, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 603}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9909", 'reported_seq': 9983, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:16.336647+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:16.336647+0000', 'last_peered': '2024-09-15T21:11:16.336647+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:16.336647+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9909, 'log_dups_size': 0, 'ondisk_log_size': 9909, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23360780, 'num_objects': 2852, 'num_object_clones': 0, 'num_object_copies': 11408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9907, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9603", 'reported_seq': 9668, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.275373+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:17.275373+0000', 'last_peered': '2024-09-15T21:11:17.275373+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:17.275373+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9603, 'log_dups_size': 0, 'ondisk_log_size': 9603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23333400, 'num_objects': 2849, 'num_object_clones': 0, 'num_object_copies': 11396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2849, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9600, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 517}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9732", 'reported_seq': 9819, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:19.187488+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:19.187488+0000', 'last_peered': '2024-09-15T21:11:19.187488+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:19.187488+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9732, 'log_dups_size': 0, 'ondisk_log_size': 9732, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23180556, 'num_objects': 2830, 'num_object_clones': 0, 'num_object_copies': 11320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2830, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9730, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 495}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9530", 'reported_seq': 9575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.354987+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:17.354987+0000', 'last_peered': '2024-09-15T21:11:17.354987+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:17.354987+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9530, 'log_dups_size': 0, 'ondisk_log_size': 9530, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23730822, 'num_objects': 2897, 'num_object_clones': 0, 'num_object_copies': 11588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2897, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9529, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 641}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9613", 'reported_seq': 10525, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:19.376670+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:19.376670+0000', 'last_peered': '2024-09-15T21:11:19.376670+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:19.376670+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9613, 'log_dups_size': 0, 'ondisk_log_size': 9613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23657094, 'num_objects': 2888, 'num_object_clones': 0, 'num_object_copies': 11552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1874, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9612, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 414, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1014}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:21.085 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:21.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9603", 'reported_seq': 9665, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:17.563489+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:17.563489+0000', 'last_peered': '2024-09-15T21:11:17.563489+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:17.563489+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9603, 'log_dups_size': 0, 'ondisk_log_size': 9603, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23172364, 'num_objects': 2829, 'num_object_clones': 0, 'num_object_copies': 11316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9601, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 550}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:21.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:21.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9751", 'reported_seq': 9839, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:19.172834+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:19.172834+0000', 'last_peered': '2024-09-15T21:11:19.172834+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:19.172834+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9751, 'log_dups_size': 0, 'ondisk_log_size': 9751, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23999756, 'num_objects': 2930, 'num_object_clones': 0, 'num_object_copies': 11720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2930, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9749, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:21.086 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:21.086 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9717", 'reported_seq': 9791, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:16.401328+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:16.401328+0000', 'last_peered': '2024-09-15T21:11:16.401328+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:16.401328+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9717, 'log_dups_size': 0, 'ondisk_log_size': 9717, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23196940, 'num_objects': 2832, 'num_object_clones': 0, 'num_object_copies': 11328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2832, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9715, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 517}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:21.087 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:21.850 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:21.851 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 32028/183684 objects degraded (17.436%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":547,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":45921,"data_bytes":376144418,"bytes_used":2865352704,"bytes_avail":383681703936,"bytes_total":386547056640,"degraded_objects":32028,"degraded_total":183684,"degraded_ratio":0.1743646697589338,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:21.876 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:21.876 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:22.476 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:22.477 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:22.510 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-15T21:11:22.109522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373441058,"num_objects":45591,"num_object_clones":0,"num_object_copies":182364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139487,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154732,"ondisk_log_size":154732,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2781700,"kb_used_data":729948,"kb_used_omap":14,"kb_used_meta":2051633,"kb_avail":374705660,"statfs":{"total":386547056640,"available":383698595840,"internally_reserved":0,"allocated":747466752,"data_stored":746774344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2100872754},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,3,21,21,23,15,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19300352,"num_objects":-2356,"num_object_clones":0,"num_object_copies":-9424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2356,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005433"},"pg_stats":[{"pgid":"1.f","version":"35'9489","reported_seq":9540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.597895+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:17.597895+0000","last_peered":"2024-09-15T21:11:17.597895+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:17.597895+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9489,"log_dups_size":0,"ondisk_log_size":9489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23453696,"num_objects":2863,"num_object_clones":0,"num_object_copies":11452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9489,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":587}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9739","reported_seq":14632,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:20.800513+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:20.800513+0000","last_peered":"2024-09-15T21:11:20.800513+0000","last_clean":"2024-09-15T21:11:20.800513+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:20.800513+0000","last_undegraded":"2024-09-15T21:11:20.800513+0000","last_fullsized":"2024-09-15T21:11:20.800513+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9739,"log_dups_size":0,"ondisk_log_size":9739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23027712,"num_objects":2811,"num_object_clones":0,"num_object_copies":11244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":405,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9542","reported_seq":9627,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.245859+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:19.245859+0000","last_peered":"2024-09-15T21:11:19.245859+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:19.245859+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9542,"log_dups_size":0,"ondisk_log_size":9542,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23673502,"num_objects":2891,"num_object_clones":0,"num_object_copies":11564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2891,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9541,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":633}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9722","reported_seq":9795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.599269+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:21.599269+0000","last_peered":"2024-09-15T21:11:21.599269+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:21.599269+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394124,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9720,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9747","reported_seq":9807,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.558361+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:17.558361+0000","last_peered":"2024-09-15T21:11:17.558361+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:17.558361+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9747,"log_dups_size":0,"ondisk_log_size":9747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23852300,"num_objects":2912,"num_object_clones":0,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9745,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9537","reported_seq":9587,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.643620+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:17.643620+0000","last_peered":"2024-09-15T21:11:17.643620+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:17.643620+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9537,"log_dups_size":0,"ondisk_log_size":9537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23257088,"num_objects":2839,"num_object_clones":0,"num_object_copies":11356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9537,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":523}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9872","reported_seq":9947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.569375+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:21.569375+0000","last_peered":"2024-09-15T21:11:21.569375+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:21.569375+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9979","reported_seq":10053,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.553997+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:21.553997+0000","last_peered":"2024-09-15T21:11:21.553997+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:21.553997+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787340,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9603","reported_seq":9668,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.275373+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:17.275373+0000","last_peered":"2024-09-15T21:11:17.275373+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:17.275373+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23333400,"num_objects":2849,"num_object_clones":0,"num_object_copies":11396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9600,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":517}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9732","reported_seq":9819,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.187488+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:19.187488+0000","last_peered":"2024-09-15T21:11:19.187488+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:19.187488+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9732,"log_dups_size":0,"ondisk_log_size":9732,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23180556,"num_objects":2830,"num_object_clones":0,"num_object_copies":11320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9730,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":495}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9500","reported_seq":14167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:17.645535+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:17.645535+0000","last_peered":"2024-09-15T21:11:17.645535+0000","last_clean":"2024-09-15T21:11:17.645535+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:17.645535+0000","last_undegraded":"2024-09-15T21:11:17.645535+0000","last_fullsized":"2024-09-15T21:11:17.645535+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9500,"log_dups_size":0,"ondisk_log_size":9500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23429120,"num_objects":2860,"num_object_clones":0,"num_object_copies":11440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9530","reported_seq":9575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.354987+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:17.354987+0000","last_peered":"2024-09-15T21:11:17.354987+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:17.354987+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9530,"log_dups_size":0,"ondisk_log_size":9530,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23730822,"num_objects":2897,"num_object_clones":0,"num_object_copies":11588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9529,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":641}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9613","reported_seq":10525,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.376670+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:19.376670+0000","last_peered":"2024-09-15T21:11:19.376670+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:19.376670+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23657094,"num_objects":2888,"num_object_clones":0,"num_object_copies":11552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9612,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1014}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9603","reported_seq":9665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:17.563489+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:17.563489+0000","last_peered":"2024-09-15T21:11:17.563489+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:17.563489+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9603,"log_dups_size":0,"ondisk_log_size":9603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23172364,"num_objects":2829,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":550}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9751","reported_seq":9839,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:19.172834+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:19.172834+0000","last_peered":"2024-09-15T21:11:19.172834+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:19.172834+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9751,"log_dups_size":0,"ondisk_log_size":9751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23999756,"num_objects":2930,"num_object_clones":0,"num_object_copies":11720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9749,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9773","reported_seq":9847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.585720+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:21.585720+0000","last_peered":"2024-09-15T21:11:21.585720+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:21.585720+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22738188,"num_objects":2776,"num_object_clones":0,"num_object_copies":11104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":373441058,"num_objects":45591,"num_object_clones":0,"num_object_copies":182364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32028,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45591,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139487,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":435,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":744579072,"data_stored":744579072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154732,"ondisk_log_size":154732,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743532,"kb_used_data":184272,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93628308,"statfs":{"total":96636764160,"available":95875387392,"internally_reserved":0,"allocated":188694528,"data_stored":188521426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,11,6,8,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745732,"kb_used_data":182820,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93626108,"statfs":{"total":96636764160,"available":95873134592,"internally_reserved":0,"allocated":187207680,"data_stored":187034578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051673,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596220,"kb_used_data":181420,"kb_used_omap":4,"kb_used_meta":414779,"kb_avail":93775620,"statfs":{"total":96636764160,"available":96026234880,"internally_reserved":0,"allocated":185774080,"data_stored":185600978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":424734415},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1,6,9,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696216,"kb_used_data":181436,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93675624,"statfs":{"total":96636764160,"available":95923838976,"internally_reserved":0,"allocated":185790464,"data_stored":185617362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":527101646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":185376768,"data_stored":185376768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":186499072,"data_stored":186499072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":187985920,"data_stored":187985920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:25.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:26.136 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:26.137 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:26.160 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-15T21:11:26.111039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367034914,"num_objects":44809,"num_object_clones":0,"num_object_copies":179236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140269,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155514,"ondisk_log_size":155514,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779324,"kb_used_data":717868,"kb_used_omap":14,"kb_used_meta":2061361,"kb_avail":374708036,"statfs":{"total":386547056640,"available":383701028864,"internally_reserved":0,"allocated":735096832,"data_stored":734404424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2110834226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,8,8,25,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004438"},"pg_stats":[{"pgid":"1.f","version":"35'9548","reported_seq":9597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.568377+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:22.568377+0000","last_peered":"2024-09-15T21:11:22.568377+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:22.568377+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22970368,"num_objects":2804,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9548,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9808","reported_seq":14697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:25.956551+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:25.956551+0000","last_peered":"2024-09-15T21:11:25.956551+0000","last_clean":"2024-09-15T21:11:25.956551+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:25.956551+0000","last_undegraded":"2024-09-15T21:11:25.956551+0000","last_fullsized":"2024-09-15T21:11:25.956551+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9613","reported_seq":9698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.371397+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:24.371397+0000","last_peered":"2024-09-15T21:11:24.371397+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:24.371397+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23091870,"num_objects":2820,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9722","reported_seq":9795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.599269+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:21.599269+0000","last_peered":"2024-09-15T21:11:21.599269+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:21.599269+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394124,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9720,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9807","reported_seq":9869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.689356+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:22.689356+0000","last_peered":"2024-09-15T21:11:22.689356+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:22.689356+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360780,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9805,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9613","reported_seq":9663,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.653944+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:22.653944+0000","last_peered":"2024-09-15T21:11:22.653944+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:22.653944+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22634496,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9872","reported_seq":9947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.569375+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:21.569375+0000","last_peered":"2024-09-15T21:11:21.569375+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:21.569375+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9979","reported_seq":10053,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.553997+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:21.553997+0000","last_peered":"2024-09-15T21:11:21.553997+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:21.553997+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787340,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9662","reported_seq":9727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.511143+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:22.511143+0000","last_peered":"2024-09-15T21:11:22.511143+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:22.511143+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22850072,"num_objects":2790,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9794","reported_seq":9881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.244163+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:24.244163+0000","last_peered":"2024-09-15T21:11:24.244163+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:24.244163+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22672652,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9571","reported_seq":14238,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:22.752443+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:22.752443+0000","last_peered":"2024-09-15T21:11:22.752443+0000","last_clean":"2024-09-15T21:11:22.752443+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:22.752443+0000","last_undegraded":"2024-09-15T21:11:22.752443+0000","last_fullsized":"2024-09-15T21:11:22.752443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9571,"log_dups_size":0,"ondisk_log_size":9571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22847488,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3682,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9607","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.200198+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:22.200198+0000","last_peered":"2024-09-15T21:11:22.200198+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:22.200198+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23100038,"num_objects":2820,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9678","reported_seq":10706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.483727+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:24.483727+0000","last_peered":"2024-09-15T21:11:24.483727+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:24.483727+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23124614,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9677,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9665","reported_seq":9727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.456684+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:22.456684+0000","last_peered":"2024-09-15T21:11:22.456684+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:22.456684+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664460,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9802","reported_seq":9890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.321455+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:24.321455+0000","last_peered":"2024-09-15T21:11:24.321455+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:24.321455+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9773","reported_seq":9847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.585720+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:21.585720+0000","last_peered":"2024-09-15T21:11:21.585720+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:21.585720+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22738188,"num_objects":2776,"num_object_clones":0,"num_object_copies":11104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367034914,"num_objects":44809,"num_object_clones":0,"num_object_copies":179236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140269,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":732160000,"data_stored":732160000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155514,"ondisk_log_size":155514,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739452,"kb_used_data":180204,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93632388,"statfs":{"total":96636764160,"available":95879565312,"internally_reserved":0,"allocated":184528896,"data_stored":184355794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729196,"kb_used_data":178776,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93642644,"statfs":{"total":96636764160,"available":95890067456,"internally_reserved":0,"allocated":183066624,"data_stored":182893522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614460,"kb_used_data":177452,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93757380,"statfs":{"total":96636764160,"available":96007557120,"internally_reserved":0,"allocated":181710848,"data_stored":181537746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696216,"kb_used_data":181436,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93675624,"statfs":{"total":96636764160,"available":95923838976,"internally_reserved":0,"allocated":185790464,"data_stored":185617362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":527101646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181198848,"data_stored":181198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182382592,"data_stored":182382592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183861248,"data_stored":183861248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:26.161 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:26.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:26.806 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:26.830 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-15T21:11:26.111039+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":367034914,"num_objects":44809,"num_object_clones":0,"num_object_copies":179236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140269,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155514,"ondisk_log_size":155514,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2779324,"kb_used_data":717868,"kb_used_omap":14,"kb_used_meta":2061361,"kb_avail":374708036,"statfs":{"total":386547056640,"available":383701028864,"internally_reserved":0,"allocated":735096832,"data_stored":734404424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2110834226},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,8,8,25,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19374080,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004438"},"pg_stats":[{"pgid":"1.f","version":"35'9548","reported_seq":9597,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.568377+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:22.568377+0000","last_peered":"2024-09-15T21:11:22.568377+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:22.568377+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9548,"log_dups_size":0,"ondisk_log_size":9548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22970368,"num_objects":2804,"num_object_clones":0,"num_object_copies":11216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9548,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9808","reported_seq":14697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:25.956551+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:25.956551+0000","last_peered":"2024-09-15T21:11:25.956551+0000","last_clean":"2024-09-15T21:11:25.956551+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:25.956551+0000","last_undegraded":"2024-09-15T21:11:25.956551+0000","last_fullsized":"2024-09-15T21:11:25.956551+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9613","reported_seq":9698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.371397+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:24.371397+0000","last_peered":"2024-09-15T21:11:24.371397+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:24.371397+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23091870,"num_objects":2820,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9722","reported_seq":9795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.599269+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:21.599269+0000","last_peered":"2024-09-15T21:11:21.599269+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:21.599269+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22394124,"num_objects":2734,"num_object_clones":0,"num_object_copies":10936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9720,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":400}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9807","reported_seq":9869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.689356+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:22.689356+0000","last_peered":"2024-09-15T21:11:22.689356+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:22.689356+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9807,"log_dups_size":0,"ondisk_log_size":9807,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23360780,"num_objects":2852,"num_object_clones":0,"num_object_copies":11408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9805,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9613","reported_seq":9663,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.653944+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:22.653944+0000","last_peered":"2024-09-15T21:11:22.653944+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:22.653944+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22634496,"num_objects":2763,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9613,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":447}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9872","reported_seq":9947,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.569375+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:21.569375+0000","last_peered":"2024-09-15T21:11:21.569375+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:21.569375+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23753996,"num_objects":2900,"num_object_clones":0,"num_object_copies":11600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'9979","reported_seq":10053,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.553997+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:21.553997+0000","last_peered":"2024-09-15T21:11:21.553997+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:21.553997+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22787340,"num_objects":2782,"num_object_clones":0,"num_object_copies":11128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":308}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9662","reported_seq":9727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.511143+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:22.511143+0000","last_peered":"2024-09-15T21:11:22.511143+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:22.511143+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9662,"log_dups_size":0,"ondisk_log_size":9662,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22850072,"num_objects":2790,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9659,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9794","reported_seq":9881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.244163+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:24.244163+0000","last_peered":"2024-09-15T21:11:24.244163+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:24.244163+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22672652,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9571","reported_seq":14238,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:22.752443+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:22.752443+0000","last_peered":"2024-09-15T21:11:22.752443+0000","last_clean":"2024-09-15T21:11:22.752443+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:22.752443+0000","last_undegraded":"2024-09-15T21:11:22.752443+0000","last_fullsized":"2024-09-15T21:11:22.752443+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9571,"log_dups_size":0,"ondisk_log_size":9571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22847488,"num_objects":2789,"num_object_clones":0,"num_object_copies":11156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3682,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9607","reported_seq":9652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.200198+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:22.200198+0000","last_peered":"2024-09-15T21:11:22.200198+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:22.200198+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9607,"log_dups_size":0,"ondisk_log_size":9607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23100038,"num_objects":2820,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9606,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":564}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9678","reported_seq":10706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.483727+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:24.483727+0000","last_peered":"2024-09-15T21:11:24.483727+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:24.483727+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23124614,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9677,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9665","reported_seq":9727,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:22.456684+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:22.456684+0000","last_peered":"2024-09-15T21:11:22.456684+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:22.456684+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9665,"log_dups_size":0,"ondisk_log_size":9665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22664460,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":488}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9802","reported_seq":9890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.321455+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:24.321455+0000","last_peered":"2024-09-15T21:11:24.321455+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:24.321455+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9773","reported_seq":9847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:21.585720+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:21.585720+0000","last_peered":"2024-09-15T21:11:21.585720+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:21.585720+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9773,"log_dups_size":0,"ondisk_log_size":9773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22738188,"num_objects":2776,"num_object_clones":0,"num_object_copies":11104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9771,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":461}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":367034914,"num_objects":44809,"num_object_clones":0,"num_object_copies":179236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44809,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140269,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":732160000,"data_stored":732160000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155514,"ondisk_log_size":155514,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739452,"kb_used_data":180204,"kb_used_omap":4,"kb_used_meta":559227,"kb_avail":93632388,"statfs":{"total":96636764160,"available":95879565312,"internally_reserved":0,"allocated":184528896,"data_stored":184355794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":572649168},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729196,"kb_used_data":178776,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93642644,"statfs":{"total":96636764160,"available":95890067456,"internally_reserved":0,"allocated":183066624,"data_stored":182893522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614460,"kb_used_data":177452,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93757380,"statfs":{"total":96636764160,"available":96007557120,"internally_reserved":0,"allocated":181710848,"data_stored":181537746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247911,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696216,"kb_used_data":181436,"kb_used_omap":4,"kb_used_meta":514747,"kb_avail":93675624,"statfs":{"total":96636764160,"available":95923838976,"internally_reserved":0,"allocated":185790464,"data_stored":185617362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":527101646},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184717312,"data_stored":184717312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181198848,"data_stored":181198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182382592,"data_stored":182382592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183861248,"data_stored":183861248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:26.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:26.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9548", 'reported_seq': 9597, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.568377+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:22.568377+0000', 'last_peered': '2024-09-15T21:11:22.568377+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:22.568377+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9548, 'log_dups_size': 0, 'ondisk_log_size': 9548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22970368, 'num_objects': 2804, 'num_object_clones': 0, 'num_object_copies': 11216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9548, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:26.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9613", 'reported_seq': 9698, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:24.371397+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:24.371397+0000', 'last_peered': '2024-09-15T21:11:24.371397+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:24.371397+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9613, 'log_dups_size': 0, 'ondisk_log_size': 9613, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23091870, 'num_objects': 2820, 'num_object_clones': 0, 'num_object_copies': 11280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2820, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9612, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 562}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:26.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9722", 'reported_seq': 9795, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:21.599269+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:21.599269+0000', 'last_peered': '2024-09-15T21:11:21.599269+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:21.599269+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9722, 'log_dups_size': 0, 'ondisk_log_size': 9722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22394124, 'num_objects': 2734, 'num_object_clones': 0, 'num_object_copies': 10936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2734, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9720, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 400}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9807", 'reported_seq': 9869, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.689356+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:22.689356+0000', 'last_peered': '2024-09-15T21:11:22.689356+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:22.689356+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9807, 'log_dups_size': 0, 'ondisk_log_size': 9807, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23360780, 'num_objects': 2852, 'num_object_clones': 0, 'num_object_copies': 11408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2852, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9805, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 541}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9613", 'reported_seq': 9663, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.653944+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:22.653944+0000', 'last_peered': '2024-09-15T21:11:22.653944+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:22.653944+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9613, 'log_dups_size': 0, 'ondisk_log_size': 9613, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22634496, 'num_objects': 2763, 'num_object_clones': 0, 'num_object_copies': 11052, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2763, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9613, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 447}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9872", 'reported_seq': 9947, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:21.569375+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:21.569375+0000', 'last_peered': '2024-09-15T21:11:21.569375+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:21.569375+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9872, 'log_dups_size': 0, 'ondisk_log_size': 9872, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23753996, 'num_objects': 2900, 'num_object_clones': 0, 'num_object_copies': 11600, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2900, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9870, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 534}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'9979", 'reported_seq': 10053, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:21.553997+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:21.553997+0000', 'last_peered': '2024-09-15T21:11:21.553997+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:21.553997+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9979, 'log_dups_size': 0, 'ondisk_log_size': 9979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22787340, 'num_objects': 2782, 'num_object_clones': 0, 'num_object_copies': 11128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9977, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 308}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:26.832 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9662", 'reported_seq': 9727, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.511143+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:22.511143+0000', 'last_peered': '2024-09-15T21:11:22.511143+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:22.511143+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9662, 'log_dups_size': 0, 'ondisk_log_size': 9662, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22850072, 'num_objects': 2790, 'num_object_clones': 0, 'num_object_copies': 11160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2790, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9659, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 458}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9794", 'reported_seq': 9881, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:24.244163+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:24.244163+0000', 'last_peered': '2024-09-15T21:11:24.244163+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:24.244163+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9794, 'log_dups_size': 0, 'ondisk_log_size': 9794, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22672652, 'num_objects': 2768, 'num_object_clones': 0, 'num_object_copies': 11072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2768, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9792, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 433}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9607", 'reported_seq': 9652, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.200198+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:22.200198+0000', 'last_peered': '2024-09-15T21:11:22.200198+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:22.200198+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9607, 'log_dups_size': 0, 'ondisk_log_size': 9607, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23100038, 'num_objects': 2820, 'num_object_clones': 0, 'num_object_copies': 11280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2820, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9606, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9678", 'reported_seq': 10706, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:24.483727+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:24.483727+0000', 'last_peered': '2024-09-15T21:11:24.483727+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:24.483727+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9678, 'log_dups_size': 0, 'ondisk_log_size': 9678, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23124614, 'num_objects': 2823, 'num_object_clones': 0, 'num_object_copies': 11292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1816, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2823, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9677, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 472, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 1007}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9665", 'reported_seq': 9727, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:22.456684+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:22.456684+0000', 'last_peered': '2024-09-15T21:11:22.456684+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:22.456684+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9665, 'log_dups_size': 0, 'ondisk_log_size': 9665, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22664460, 'num_objects': 2767, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9663, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 488}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:26.833 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9802", 'reported_seq': 9890, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:24.321455+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:24.321455+0000', 'last_peered': '2024-09-15T21:11:24.321455+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:24.321455+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9802, 'log_dups_size': 0, 'ondisk_log_size': 9802, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23581964, 'num_objects': 2879, 'num_object_clones': 0, 'num_object_copies': 11516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9800, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 598}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:26.834 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:26.834 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9773", 'reported_seq': 9847, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:21.585720+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:21.585720+0000', 'last_peered': '2024-09-15T21:11:21.585720+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:21.585720+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9773, 'log_dups_size': 0, 'ondisk_log_size': 9773, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22738188, 'num_objects': 2776, 'num_object_clones': 0, 'num_object_copies': 11104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2776, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9771, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 461}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:26.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:27.577 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:27.577 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31970/179236 objects degraded (17.837%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":553,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":44809,"data_bytes":367034914,"bytes_used":2846027776,"bytes_avail":383701028864,"bytes_total":386547056640,"degraded_objects":31970,"degraded_total":179236,"degraded_ratio":0.17836818496284232,"recovering_objects_per_sec":13,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":166,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":197},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:27.601 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:27.601 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:28.211 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:28.212 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:28.239 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-15T21:11:28.112000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":361456162,"num_objects":44128,"num_object_clones":0,"num_object_copies":176512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140950,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156195,"ondisk_log_size":156195,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2796920,"kb_used_data":709096,"kb_used_omap":14,"kb_used_meta":2087729,"kb_avail":374690440,"statfs":{"total":386547056640,"available":383683010560,"internally_reserved":0,"allocated":726114304,"data_stored":725421896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2137835058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,6,8,23,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22372352,"num_objects":-2731,"num_object_clones":0,"num_object_copies":-10924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004528"},"pg_stats":[{"pgid":"1.f","version":"35'9602","reported_seq":9653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.625992+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:27.625992+0000","last_peered":"2024-09-15T21:11:27.625992+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:27.625992+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22528000,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9808","reported_seq":14697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:25.956551+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:25.956551+0000","last_peered":"2024-09-15T21:11:25.956551+0000","last_clean":"2024-09-15T21:11:25.956551+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:25.956551+0000","last_undegraded":"2024-09-15T21:11:25.956551+0000","last_fullsized":"2024-09-15T21:11:25.956551+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9613","reported_seq":9698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.371397+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:24.371397+0000","last_peered":"2024-09-15T21:11:24.371397+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:24.371397+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9613,"log_dups_size":0,"ondisk_log_size":9613,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23091870,"num_objects":2820,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":562}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9769","reported_seq":9844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.287160+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:26.287160+0000","last_peered":"2024-09-15T21:11:26.287160+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:26.287160+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22009100,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9871","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.659552+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:27.659552+0000","last_peered":"2024-09-15T21:11:27.659552+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:27.659552+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22836492,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9683","reported_seq":9733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.593099+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:27.593099+0000","last_peered":"2024-09-15T21:11:27.593099+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:27.593099+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22061056,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9931","reported_seq":10007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.728195+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:26.728195+0000","last_peered":"2024-09-15T21:11:26.728195+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:26.728195+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23270668,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10041","reported_seq":10114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.756153+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:26.756153+0000","last_peered":"2024-09-15T21:11:26.756153+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:26.756153+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279436,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.590027+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:27.590027+0000","last_peered":"2024-09-15T21:11:27.590027+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:27.590027+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22358552,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9794","reported_seq":9881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.244163+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:24.244163+0000","last_peered":"2024-09-15T21:11:24.244163+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:24.244163+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9794,"log_dups_size":0,"ondisk_log_size":9794,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22672652,"num_objects":2768,"num_object_clones":0,"num_object_copies":11072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9792,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":433}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9639","reported_seq":14306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:27.649397+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:27.649397+0000","last_peered":"2024-09-15T21:11:27.649397+0000","last_clean":"2024-09-15T21:11:27.649397+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:27.649397+0000","last_undegraded":"2024-09-15T21:11:27.649397+0000","last_fullsized":"2024-09-15T21:11:27.649397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9639,"log_dups_size":0,"ondisk_log_size":9639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9679","reported_seq":9724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.537635+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:27.537635+0000","last_peered":"2024-09-15T21:11:27.537635+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:27.537635+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22510214,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9678","reported_seq":10706,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.483727+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:24.483727+0000","last_peered":"2024-09-15T21:11:24.483727+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:24.483727+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23124614,"num_objects":2823,"num_object_clones":0,"num_object_copies":11292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9677,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":1007}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9727","reported_seq":9789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.686457+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:27.686457+0000","last_peered":"2024-09-15T21:11:27.686457+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:27.686457+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22156556,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9802","reported_seq":9890,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:24.321455+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:24.321455+0000","last_peered":"2024-09-15T21:11:24.321455+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:24.321455+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9802,"log_dups_size":0,"ondisk_log_size":9802,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23581964,"num_objects":2879,"num_object_clones":0,"num_object_copies":11516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9800,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":598}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9836","reported_seq":9910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.760152+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:26.760152+0000","last_peered":"2024-09-15T21:11:26.760152+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:26.760152+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22222092,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":361456162,"num_objects":44128,"num_object_clones":0,"num_object_copies":176512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140950,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":723869696,"data_stored":723869696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156195,"ondisk_log_size":156195,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739600,"kb_used_data":176192,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93632240,"statfs":{"total":96636764160,"available":95879413760,"internally_reserved":0,"allocated":180420608,"data_stored":180247506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":576909008},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729196,"kb_used_data":178776,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93642644,"statfs":{"total":96636764160,"available":95890067456,"internally_reserved":0,"allocated":183066624,"data_stored":182893522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614460,"kb_used_data":177452,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93757380,"statfs":{"total":96636764160,"available":96007557120,"internally_reserved":0,"allocated":181710848,"data_stored":181537746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":176676,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":180916224,"data_stored":180743122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":549842638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180510720,"data_stored":180510720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181198848,"data_stored":181198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":182382592,"data_stored":182382592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":179777536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:31.241 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:31.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:31.881 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:31.906 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-15T21:11:30.112900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359498274,"num_objects":43889,"num_object_clones":0,"num_object_copies":175556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156434,"ondisk_log_size":156434,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2815076,"kb_used_data":705040,"kb_used_omap":14,"kb_used_meta":2109937,"kb_avail":374672284,"statfs":{"total":386547056640,"available":383664418816,"internally_reserved":0,"allocated":721960960,"data_stored":721268552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2160576050},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,5,6,23,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18546688,"num_objects":-2264,"num_object_clones":0,"num_object_copies":-9056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004741"},"pg_stats":[{"pgid":"1.f","version":"35'9602","reported_seq":9653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.625992+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:27.625992+0000","last_peered":"2024-09-15T21:11:27.625992+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:27.625992+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22528000,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9808","reported_seq":14697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:25.956551+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:25.956551+0000","last_peered":"2024-09-15T21:11:25.956551+0000","last_clean":"2024-09-15T21:11:25.956551+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:25.956551+0000","last_undegraded":"2024-09-15T21:11:25.956551+0000","last_fullsized":"2024-09-15T21:11:25.956551+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9808,"log_dups_size":0,"ondisk_log_size":9808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22462464,"num_objects":2742,"num_object_clones":0,"num_object_copies":10968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9666","reported_seq":9751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.205353+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:29.205353+0000","last_peered":"2024-09-15T21:11:29.205353+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:29.205353+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657694,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9665,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9769","reported_seq":9844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.287160+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:26.287160+0000","last_peered":"2024-09-15T21:11:26.287160+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:26.287160+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9769,"log_dups_size":0,"ondisk_log_size":9769,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22009100,"num_objects":2687,"num_object_clones":0,"num_object_copies":10748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9767,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":353}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9871","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.659552+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:27.659552+0000","last_peered":"2024-09-15T21:11:27.659552+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:27.659552+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22836492,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9683","reported_seq":9733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.593099+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:27.593099+0000","last_peered":"2024-09-15T21:11:27.593099+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:27.593099+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22061056,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9931","reported_seq":10007,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.728195+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:26.728195+0000","last_peered":"2024-09-15T21:11:26.728195+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:26.728195+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9931,"log_dups_size":0,"ondisk_log_size":9931,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23270668,"num_objects":2841,"num_object_clones":0,"num_object_copies":11364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9929,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":475}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10041","reported_seq":10114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.756153+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:26.756153+0000","last_peered":"2024-09-15T21:11:26.756153+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:26.756153+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22279436,"num_objects":2720,"num_object_clones":0,"num_object_copies":10880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":246}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.590027+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:27.590027+0000","last_peered":"2024-09-15T21:11:27.590027+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:27.590027+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22358552,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9864","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.423008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:29.423008+0000","last_peered":"2024-09-15T21:11:29.423008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:29.423008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22099212,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9639","reported_seq":14306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:27.649397+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:27.649397+0000","last_peered":"2024-09-15T21:11:27.649397+0000","last_clean":"2024-09-15T21:11:27.649397+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:27.649397+0000","last_undegraded":"2024-09-15T21:11:27.649397+0000","last_fullsized":"2024-09-15T21:11:27.649397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9639,"log_dups_size":0,"ondisk_log_size":9639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9679","reported_seq":9724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.537635+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:27.537635+0000","last_peered":"2024-09-15T21:11:27.537635+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:27.537635+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22510214,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9743","reported_seq":10861,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.464255+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:29.464255+0000","last_peered":"2024-09-15T21:11:29.464255+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:29.464255+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9743,"log_dups_size":0,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22592134,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9727","reported_seq":9789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.686457+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:27.686457+0000","last_peered":"2024-09-15T21:11:27.686457+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:27.686457+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22156556,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9853","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.405517+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:29.405517+0000","last_peered":"2024-09-15T21:11:29.405517+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:29.405517+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23164172,"num_objects":2828,"num_object_clones":0,"num_object_copies":11312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9836","reported_seq":9910,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:26.760152+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:26.760152+0000","last_peered":"2024-09-15T21:11:26.760152+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:26.760152+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9836,"log_dups_size":0,"ondisk_log_size":9836,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22222092,"num_objects":2713,"num_object_clones":0,"num_object_copies":10852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9834,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":398}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":359498274,"num_objects":43889,"num_object_clones":0,"num_object_copies":175556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":719781888,"data_stored":719781888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156434,"ondisk_log_size":156434,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739600,"kb_used_data":176192,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93632240,"statfs":{"total":96636764160,"available":95879413760,"internally_reserved":0,"allocated":180420608,"data_stored":180247506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":576909008},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747352,"kb_used_data":174720,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93624488,"statfs":{"total":96636764160,"available":95871475712,"internally_reserved":0,"allocated":178913280,"data_stored":178740178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051674,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":614460,"kb_used_data":177452,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93757380,"statfs":{"total":96636764160,"available":96007557120,"internally_reserved":0,"allocated":181710848,"data_stored":181537746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247912,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":176676,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":180916224,"data_stored":180743122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":549842638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180510720,"data_stored":180510720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":181198848,"data_stored":181198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178294784,"data_stored":178294784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":179777536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:31.907 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:32.533 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:32.533 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:32.559 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-15T21:11:32.113736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356901410,"num_objects":43572,"num_object_clones":0,"num_object_copies":174288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141506,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156751,"ondisk_log_size":156751,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853056,"kb_used_data":697196,"kb_used_omap":14,"kb_used_meta":2155761,"kb_avail":374634304,"statfs":{"total":386547056640,"available":383625527296,"internally_reserved":0,"allocated":713928704,"data_stored":713236296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2207499826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,7,4,7,22,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19243008,"num_objects":-2349,"num_object_clones":0,"num_object_copies":-9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004891"},"pg_stats":[{"pgid":"1.f","version":"35'9602","reported_seq":9653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.625992+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:27.625992+0000","last_peered":"2024-09-15T21:11:27.625992+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:27.625992+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22528000,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9881","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:30.985145+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:30.985145+0000","last_peered":"2024-09-15T21:11:30.985145+0000","last_clean":"2024-09-15T21:11:30.985145+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:30.985145+0000","last_undegraded":"2024-09-15T21:11:30.985145+0000","last_fullsized":"2024-09-15T21:11:30.985145+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9666","reported_seq":9751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.205353+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:29.205353+0000","last_peered":"2024-09-15T21:11:29.205353+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:29.205353+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657694,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9665,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9839","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.763145+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:31.763145+0000","last_peered":"2024-09-15T21:11:31.763145+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:31.763145+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9839,"log_dups_size":0,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9871","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.659552+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:27.659552+0000","last_peered":"2024-09-15T21:11:27.659552+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:27.659552+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22836492,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9683","reported_seq":9733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.593099+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:27.593099+0000","last_peered":"2024-09-15T21:11:27.593099+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:27.593099+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22061056,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9991","reported_seq":10067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.808117+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:31.808117+0000","last_peered":"2024-09-15T21:11:31.808117+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:31.808117+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779148,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10099","reported_seq":10172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.684309+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:31.684309+0000","last_peered":"2024-09-15T21:11:31.684309+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:31.684309+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804300,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.590027+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:27.590027+0000","last_peered":"2024-09-15T21:11:27.590027+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:27.590027+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22358552,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9864","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.423008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:29.423008+0000","last_peered":"2024-09-15T21:11:29.423008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:29.423008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22099212,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9639","reported_seq":14306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:27.649397+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:27.649397+0000","last_peered":"2024-09-15T21:11:27.649397+0000","last_clean":"2024-09-15T21:11:27.649397+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:27.649397+0000","last_undegraded":"2024-09-15T21:11:27.649397+0000","last_fullsized":"2024-09-15T21:11:27.649397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9639,"log_dups_size":0,"ondisk_log_size":9639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9679","reported_seq":9724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.537635+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:27.537635+0000","last_peered":"2024-09-15T21:11:27.537635+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:27.537635+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22510214,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9743","reported_seq":10861,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.464255+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:29.464255+0000","last_peered":"2024-09-15T21:11:29.464255+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:29.464255+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9743,"log_dups_size":0,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22592134,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9727","reported_seq":9789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.686457+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:27.686457+0000","last_peered":"2024-09-15T21:11:27.686457+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:27.686457+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22156556,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9853","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.405517+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:29.405517+0000","last_peered":"2024-09-15T21:11:29.405517+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:29.405517+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23164172,"num_objects":2828,"num_object_clones":0,"num_object_copies":11312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9892","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.658575+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:31.658575+0000","last_peered":"2024-09-15T21:11:31.658575+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:31.658575+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9890,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356901410,"num_objects":43572,"num_object_clones":0,"num_object_copies":174288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141506,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":711499776,"data_stored":711499776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156751,"ondisk_log_size":156751,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739600,"kb_used_data":176192,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93632240,"statfs":{"total":96636764160,"available":95879413760,"internally_reserved":0,"allocated":180420608,"data_stored":180247506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":576909008},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747352,"kb_used_data":174720,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93624488,"statfs":{"total":96636764160,"available":95871475712,"internally_reserved":0,"allocated":178913280,"data_stored":178740178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598060,"kb_used_data":173532,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93773780,"statfs":{"total":96636764160,"available":96024350720,"internally_reserved":0,"allocated":177696768,"data_stored":177523666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768044,"kb_used_data":172752,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93603796,"statfs":{"total":96636764160,"available":95850287104,"internally_reserved":0,"allocated":176898048,"data_stored":176724946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176353280,"data_stored":176353280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177074176,"data_stored":177074176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178294784,"data_stored":178294784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":179777536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9602", 'reported_seq': 9653, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.625992+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:27.625992+0000', 'last_peered': '2024-09-15T21:11:27.625992+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:27.625992+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9602, 'log_dups_size': 0, 'ondisk_log_size': 9602, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22528000, 'num_objects': 2750, 'num_object_clones': 0, 'num_object_copies': 11000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9602, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 474}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9666", 'reported_seq': 9751, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:29.205353+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:29.205353+0000', 'last_peered': '2024-09-15T21:11:29.205353+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:29.205353+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9666, 'log_dups_size': 0, 'ondisk_log_size': 9666, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22657694, 'num_objects': 2767, 'num_object_clones': 0, 'num_object_copies': 11068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2767, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9665, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 509}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:32.560 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9839", 'reported_seq': 9914, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:31.763145+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:31.763145+0000', 'last_peered': '2024-09-15T21:11:31.763145+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:31.763145+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9839, 'log_dups_size': 0, 'ondisk_log_size': 9839, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21435660, 'num_objects': 2617, 'num_object_clones': 0, 'num_object_copies': 10468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2617, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9837, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 283}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'9871", 'reported_seq': 9933, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.659552+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:27.659552+0000', 'last_peered': '2024-09-15T21:11:27.659552+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:27.659552+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9871, 'log_dups_size': 0, 'ondisk_log_size': 9871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22836492, 'num_objects': 2788, 'num_object_clones': 0, 'num_object_copies': 11152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2788, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9869, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 477}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9683", 'reported_seq': 9733, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.593099+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:27.593099+0000', 'last_peered': '2024-09-15T21:11:27.593099+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:27.593099+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9683, 'log_dups_size': 0, 'ondisk_log_size': 9683, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22061056, 'num_objects': 2693, 'num_object_clones': 0, 'num_object_copies': 10772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2693, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9683, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 377}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'9991", 'reported_seq': 10067, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:31.808117+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:31.808117+0000', 'last_peered': '2024-09-15T21:11:31.808117+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:31.808117+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9991, 'log_dups_size': 0, 'ondisk_log_size': 9991, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22779148, 'num_objects': 2781, 'num_object_clones': 0, 'num_object_copies': 11124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2781, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9989, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 415}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10099", 'reported_seq': 10172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:31.684309+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:31.684309+0000', 'last_peered': '2024-09-15T21:11:31.684309+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:31.684309+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21804300, 'num_objects': 2662, 'num_object_clones': 0, 'num_object_copies': 10648, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2662, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10097, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 188}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:32.561 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9722", 'reported_seq': 9786, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.590027+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:27.590027+0000', 'last_peered': '2024-09-15T21:11:27.590027+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:27.590027+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9722, 'log_dups_size': 0, 'ondisk_log_size': 9722, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22358552, 'num_objects': 2730, 'num_object_clones': 0, 'num_object_copies': 10920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2730, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9719, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 398}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9864", 'reported_seq': 9951, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:29.423008+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:29.423008+0000', 'last_peered': '2024-09-15T21:11:29.423008+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:29.423008+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9864, 'log_dups_size': 0, 'ondisk_log_size': 9864, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22099212, 'num_objects': 2698, 'num_object_clones': 0, 'num_object_copies': 10792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9862, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 363}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9679", 'reported_seq': 9724, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.537635+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:27.537635+0000', 'last_peered': '2024-09-15T21:11:27.537635+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:27.537635+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9679, 'log_dups_size': 0, 'ondisk_log_size': 9679, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22510214, 'num_objects': 2748, 'num_object_clones': 0, 'num_object_copies': 10992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9678, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 492}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9743", 'reported_seq': 10861, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:29.464255+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:29.464255+0000', 'last_peered': '2024-09-15T21:11:29.464255+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:29.464255+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9743, 'log_dups_size': 0, 'ondisk_log_size': 9743, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22592134, 'num_objects': 2758, 'num_object_clones': 0, 'num_object_copies': 11032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1771, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9742, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 517, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 987}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9727", 'reported_seq': 9789, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:27.686457+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:27.686457+0000', 'last_peered': '2024-09-15T21:11:27.686457+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:27.686457+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9727, 'log_dups_size': 0, 'ondisk_log_size': 9727, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22156556, 'num_objects': 2705, 'num_object_clones': 0, 'num_object_copies': 10820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2705, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9725, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 426}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9853", 'reported_seq': 9941, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:29.405517+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:29.405517+0000', 'last_peered': '2024-09-15T21:11:29.405517+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:29.405517+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9853, 'log_dups_size': 0, 'ondisk_log_size': 9853, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 23164172, 'num_objects': 2828, 'num_object_clones': 0, 'num_object_copies': 11312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2828, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9851, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 547}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:32.562 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9892", 'reported_seq': 9966, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:31.658575+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:31.658575+0000', 'last_peered': '2024-09-15T21:11:31.658575+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:31.658575+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9892, 'log_dups_size': 0, 'ondisk_log_size': 9892, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21763340, 'num_objects': 2657, 'num_object_clones': 0, 'num_object_copies': 10628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9890, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 342}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:32.563 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:33.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:33.323 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31925/174288 objects degraded (18.317%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":559,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":43572,"data_bytes":356901410,"bytes_used":2921529344,"bytes_avail":383625527296,"bytes_total":386547056640,"degraded_objects":31925,"degraded_total":174288,"degraded_ratio":0.18317382722849537,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":195},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:33.356 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:33.356 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:33.941 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:33.941 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:33.965 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-15T21:11:32.113736+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356901410,"num_objects":43572,"num_object_clones":0,"num_object_copies":174288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141506,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156751,"ondisk_log_size":156751,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2853056,"kb_used_data":697196,"kb_used_omap":14,"kb_used_meta":2155761,"kb_avail":374634304,"statfs":{"total":386547056640,"available":383625527296,"internally_reserved":0,"allocated":713928704,"data_stored":713236296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2207499826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,7,4,7,22,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19243008,"num_objects":-2349,"num_object_clones":0,"num_object_copies":-9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-103,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2349,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004891"},"pg_stats":[{"pgid":"1.f","version":"35'9602","reported_seq":9653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.625992+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:27.625992+0000","last_peered":"2024-09-15T21:11:27.625992+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:27.625992+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9602,"log_dups_size":0,"ondisk_log_size":9602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22528000,"num_objects":2750,"num_object_clones":0,"num_object_copies":11000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9602,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":474}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9881","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:30.985145+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:30.985145+0000","last_peered":"2024-09-15T21:11:30.985145+0000","last_clean":"2024-09-15T21:11:30.985145+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:30.985145+0000","last_undegraded":"2024-09-15T21:11:30.985145+0000","last_fullsized":"2024-09-15T21:11:30.985145+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9666","reported_seq":9751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.205353+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:29.205353+0000","last_peered":"2024-09-15T21:11:29.205353+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:29.205353+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9666,"log_dups_size":0,"ondisk_log_size":9666,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22657694,"num_objects":2767,"num_object_clones":0,"num_object_copies":11068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2767,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9665,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":509}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9839","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.763145+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:31.763145+0000","last_peered":"2024-09-15T21:11:31.763145+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:31.763145+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9839,"log_dups_size":0,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9871","reported_seq":9933,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.659552+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:27.659552+0000","last_peered":"2024-09-15T21:11:27.659552+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:27.659552+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22836492,"num_objects":2788,"num_object_clones":0,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9869,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":477}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9683","reported_seq":9733,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.593099+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:27.593099+0000","last_peered":"2024-09-15T21:11:27.593099+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:27.593099+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9683,"log_dups_size":0,"ondisk_log_size":9683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22061056,"num_objects":2693,"num_object_clones":0,"num_object_copies":10772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9683,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":377}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9991","reported_seq":10067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.808117+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:31.808117+0000","last_peered":"2024-09-15T21:11:31.808117+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:31.808117+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779148,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10099","reported_seq":10172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.684309+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:31.684309+0000","last_peered":"2024-09-15T21:11:31.684309+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:31.684309+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804300,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9722","reported_seq":9786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.590027+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:27.590027+0000","last_peered":"2024-09-15T21:11:27.590027+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:27.590027+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22358552,"num_objects":2730,"num_object_clones":0,"num_object_copies":10920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9864","reported_seq":9951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.423008+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:29.423008+0000","last_peered":"2024-09-15T21:11:29.423008+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:29.423008+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22099212,"num_objects":2698,"num_object_clones":0,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9862,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":363}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9639","reported_seq":14306,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:27.649397+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:27.649397+0000","last_peered":"2024-09-15T21:11:27.649397+0000","last_clean":"2024-09-15T21:11:27.649397+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:27.649397+0000","last_undegraded":"2024-09-15T21:11:27.649397+0000","last_fullsized":"2024-09-15T21:11:27.649397+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9639,"log_dups_size":0,"ondisk_log_size":9639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22290432,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3750,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9679","reported_seq":9724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.537635+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:27.537635+0000","last_peered":"2024-09-15T21:11:27.537635+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:27.537635+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9679,"log_dups_size":0,"ondisk_log_size":9679,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22510214,"num_objects":2748,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":492}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9743","reported_seq":10861,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.464255+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:29.464255+0000","last_peered":"2024-09-15T21:11:29.464255+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:29.464255+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9743,"log_dups_size":0,"ondisk_log_size":9743,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22592134,"num_objects":2758,"num_object_clones":0,"num_object_copies":11032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9742,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":987}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9727","reported_seq":9789,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:27.686457+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:27.686457+0000","last_peered":"2024-09-15T21:11:27.686457+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:27.686457+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9727,"log_dups_size":0,"ondisk_log_size":9727,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22156556,"num_objects":2705,"num_object_clones":0,"num_object_copies":10820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9725,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":426}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9853","reported_seq":9941,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:29.405517+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:29.405517+0000","last_peered":"2024-09-15T21:11:29.405517+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:29.405517+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9853,"log_dups_size":0,"ondisk_log_size":9853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23164172,"num_objects":2828,"num_object_clones":0,"num_object_copies":11312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":547}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9892","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.658575+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:31.658575+0000","last_peered":"2024-09-15T21:11:31.658575+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:31.658575+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9890,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":356901410,"num_objects":43572,"num_object_clones":0,"num_object_copies":174288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43572,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141506,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":711499776,"data_stored":711499776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156751,"ondisk_log_size":156751,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739600,"kb_used_data":176192,"kb_used_omap":4,"kb_used_meta":563387,"kb_avail":93632240,"statfs":{"total":96636764160,"available":95879413760,"internally_reserved":0,"allocated":180420608,"data_stored":180247506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":576909008},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,15,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747352,"kb_used_data":174720,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93624488,"statfs":{"total":96636764160,"available":95871475712,"internally_reserved":0,"allocated":178913280,"data_stored":178740178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598060,"kb_used_data":173532,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93773780,"statfs":{"total":96636764160,"available":96024350720,"internally_reserved":0,"allocated":177696768,"data_stored":177523666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768044,"kb_used_data":172752,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93603796,"statfs":{"total":96636764160,"available":95850287104,"internally_reserved":0,"allocated":176898048,"data_stored":176724946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176353280,"data_stored":176353280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177074176,"data_stored":177074176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":178294784,"data_stored":178294784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179777536,"data_stored":179777536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:36.967 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:37.581 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:37.581 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:37.612 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-15T21:11:36.115383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":350970402,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142230,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157475,"ondisk_log_size":157475,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2859052,"kb_used_data":689288,"kb_used_omap":14,"kb_used_meta":2169649,"kb_avail":374628308,"statfs":{"total":386547056640,"available":383619387392,"internally_reserved":0,"allocated":705830912,"data_stored":705138504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2221721138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,10,3,9,20,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18669568,"num_objects":-2279,"num_object_clones":0,"num_object_copies":-9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005228"},"pg_stats":[{"pgid":"1.f","version":"35'9678","reported_seq":9729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.799097+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:32.799097+0000","last_peered":"2024-09-15T21:11:32.799097+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:32.799097+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9678,"log_dups_size":0,"ondisk_log_size":9678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21905408,"num_objects":2674,"num_object_clones":0,"num_object_copies":10696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9678,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":398}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9881","reported_seq":14774,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:30.985145+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:30.985145+0000","last_peered":"2024-09-15T21:11:30.985145+0000","last_clean":"2024-09-15T21:11:30.985145+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:30.985145+0000","last_undegraded":"2024-09-15T21:11:30.985145+0000","last_fullsized":"2024-09-15T21:11:30.985145+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9881,"log_dups_size":0,"ondisk_log_size":9881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21864448,"num_objects":2669,"num_object_clones":0,"num_object_copies":10676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9748","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.593583+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:34.593583+0000","last_peered":"2024-09-15T21:11:34.593583+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:34.593583+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21985950,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9839","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.763145+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:31.763145+0000","last_peered":"2024-09-15T21:11:31.763145+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:31.763145+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9839,"log_dups_size":0,"ondisk_log_size":9839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21435660,"num_objects":2617,"num_object_clones":0,"num_object_copies":10468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":283}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'9938","reported_seq":10000,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.822032+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:32.822032+0000","last_peered":"2024-09-15T21:11:32.822032+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:32.822032+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22287628,"num_objects":2721,"num_object_clones":0,"num_object_copies":10884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9936,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":410}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9738","reported_seq":9788,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.681778+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:32.681778+0000","last_peered":"2024-09-15T21:11:32.681778+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:32.681778+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9738,"log_dups_size":0,"ondisk_log_size":9738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21610496,"num_objects":2638,"num_object_clones":0,"num_object_copies":10552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9738,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'9991","reported_seq":10067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.808117+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:31.808117+0000","last_peered":"2024-09-15T21:11:31.808117+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:31.808117+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9991,"log_dups_size":0,"ondisk_log_size":9991,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22779148,"num_objects":2781,"num_object_clones":0,"num_object_copies":11124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9989,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":415}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10099","reported_seq":10172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.684309+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:31.684309+0000","last_peered":"2024-09-15T21:11:31.684309+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:31.684309+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21804300,"num_objects":2662,"num_object_clones":0,"num_object_copies":10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":188}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9788","reported_seq":9853,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.799210+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:32.799210+0000","last_peered":"2024-09-15T21:11:32.799210+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:32.799210+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9788,"log_dups_size":0,"ondisk_log_size":9788,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21817880,"num_objects":2664,"num_object_clones":0,"num_object_copies":10656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9785,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":332}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9922","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.482533+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:34.482533+0000","last_peered":"2024-09-15T21:11:34.482533+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:34.482533+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21624076,"num_objects":2640,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9702","reported_seq":14371,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:32.744628+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:32.744628+0000","last_peered":"2024-09-15T21:11:32.744628+0000","last_clean":"2024-09-15T21:11:32.744628+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:32.744628+0000","last_undegraded":"2024-09-15T21:11:32.744628+0000","last_fullsized":"2024-09-15T21:11:32.744628+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21774336,"num_objects":2658,"num_object_clones":0,"num_object_copies":10632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3813,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9748","reported_seq":9793,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.597020+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:32.597020+0000","last_peered":"2024-09-15T21:11:32.597020+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:32.597020+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21944966,"num_objects":2679,"num_object_clones":0,"num_object_copies":10716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9747,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":423}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9805","reported_seq":11022,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.582315+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:34.582315+0000","last_peered":"2024-09-15T21:11:34.582315+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:34.582315+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084230,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9785","reported_seq":9847,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:32.674706+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:32.674706+0000","last_peered":"2024-09-15T21:11:32.674706+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:32.674706+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9785,"log_dups_size":0,"ondisk_log_size":9785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21681420,"num_objects":2647,"num_object_clones":0,"num_object_copies":10588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9783,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":368}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9921","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.580229+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:34.580229+0000","last_peered":"2024-09-15T21:11:34.580229+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:34.580229+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607116,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9892","reported_seq":9966,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:31.658575+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:31.658575+0000","last_peered":"2024-09-15T21:11:31.658575+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:31.658575+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9892,"log_dups_size":0,"ondisk_log_size":9892,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9890,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":342}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":350970402,"num_objects":42848,"num_object_clones":0,"num_object_copies":171392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42848,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142230,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":703221760,"data_stored":703221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":157475,"ondisk_log_size":157475,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":745272,"kb_used_data":172132,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93626568,"statfs":{"total":96636764160,"available":95873605632,"internally_reserved":0,"allocated":176263168,"data_stored":176090066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,0,0,8,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747676,"kb_used_data":170872,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93624164,"statfs":{"total":96636764160,"available":95871143936,"internally_reserved":0,"allocated":174972928,"data_stored":174799826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051675,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598060,"kb_used_data":173532,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93773780,"statfs":{"total":96636764160,"available":96024350720,"internally_reserved":0,"allocated":177696768,"data_stored":177523666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,3,3,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247913,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768044,"kb_used_data":172752,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93603796,"statfs":{"total":96636764160,"available":95850287104,"internally_reserved":0,"allocated":176898048,"data_stored":176724946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176353280,"data_stored":176353280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":177074176,"data_stored":177074176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174215168,"data_stored":174215168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":175579136,"data_stored":175579136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:37.613 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:38.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:38.255 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:38.279 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-15T21:11:38.116272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344793634,"num_objects":42094,"num_object_clones":0,"num_object_copies":168376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142984,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158129,"ondisk_log_size":158129,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2855612,"kb_used_data":677544,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374631748,"statfs":{"total":386547056640,"available":383622909952,"internally_reserved":0,"allocated":693805056,"data_stored":693112648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,11,5,7,18,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22241280,"num_objects":-2715,"num_object_clones":0,"num_object_copies":-10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005233"},"pg_stats":[{"pgid":"1.f","version":"35'9755","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.052650+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:38.052650+0000","last_peered":"2024-09-15T21:11:38.052650+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:38.052650+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9932","reported_seq":14825,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:36.236919+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:36.236919+0000","last_peered":"2024-09-15T21:11:36.236919+0000","last_clean":"2024-09-15T21:11:36.236919+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:36.236919+0000","last_undegraded":"2024-09-15T21:11:36.236919+0000","last_fullsized":"2024-09-15T21:11:36.236919+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21446656,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9748","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.593583+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:34.593583+0000","last_peered":"2024-09-15T21:11:34.593583+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:34.593583+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21985950,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9904","reported_seq":9979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.721355+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:36.721355+0000","last_peered":"2024-09-15T21:11:36.721355+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:36.721355+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20903180,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.943656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:37.943656+0000","last_peered":"2024-09-15T21:11:37.943656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:37.943656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9798","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.959275+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:37.959275+0000","last_peered":"2024-09-15T21:11:37.959275+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:37.959275+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10057","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.660909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:36.660909+0000","last_peered":"2024-09-15T21:11:36.660909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:36.660909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22238476,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":349}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10152","reported_seq":10227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.820877+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:36.820877+0000","last_peered":"2024-09-15T21:11:36.820877+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:36.820877+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370124,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9851","reported_seq":9915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.010614+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:38.010614+0000","last_peered":"2024-09-15T21:11:38.010614+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:38.010614+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21301784,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9922","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.482533+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:34.482533+0000","last_peered":"2024-09-15T21:11:34.482533+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:34.482533+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21624076,"num_objects":2640,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9774","reported_seq":14443,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:38.071344+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:38.071344+0000","last_peered":"2024-09-15T21:11:38.071344+0000","last_clean":"2024-09-15T21:11:38.071344+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:38.071344+0000","last_undegraded":"2024-09-15T21:11:38.071344+0000","last_fullsized":"2024-09-15T21:11:38.071344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21184512,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9813","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.917301+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:37.917301+0000","last_peered":"2024-09-15T21:11:37.917301+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:37.917301+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412486,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9805","reported_seq":11022,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.582315+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:34.582315+0000","last_peered":"2024-09-15T21:11:34.582315+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:34.582315+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084230,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9850","reported_seq":9912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.627131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:37.627131+0000","last_peered":"2024-09-15T21:11:37.627131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:37.627131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9921","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.580229+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:34.580229+0000","last_peered":"2024-09-15T21:11:34.580229+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:34.580229+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607116,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9945","reported_seq":10019,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.598329+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:36.598329+0000","last_peered":"2024-09-15T21:11:36.598329+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:36.598329+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21329164,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":344793634,"num_objects":42094,"num_object_clones":0,"num_object_copies":168376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142984,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":690720768,"data_stored":690720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158129,"ondisk_log_size":158129,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741288,"kb_used_data":168152,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93630552,"statfs":{"total":96636764160,"available":95877685248,"internally_reserved":0,"allocated":172187648,"data_stored":172014546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,1,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747676,"kb_used_data":170872,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93624164,"statfs":{"total":96636764160,"available":95871143936,"internally_reserved":0,"allocated":174972928,"data_stored":174799826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598292,"kb_used_data":169604,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93773548,"statfs":{"total":96636764160,"available":96024113152,"internally_reserved":0,"allocated":173674496,"data_stored":173501394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768356,"kb_used_data":168916,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93603484,"statfs":{"total":96636764160,"available":95849967616,"internally_reserved":0,"allocated":172969984,"data_stored":172796882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,3,0,2,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172290048,"data_stored":172290048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172875776,"data_stored":172875776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174215168,"data_stored":174215168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171339776,"data_stored":171339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9755", 'reported_seq': 9805, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:38.052650+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:38.052650+0000', 'last_peered': '2024-09-15T21:11:38.052650+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:38.052650+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9755, 'log_dups_size': 0, 'ondisk_log_size': 9755, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21274624, 'num_objects': 2597, 'num_object_clones': 0, 'num_object_copies': 10388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9755, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 321}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9748", 'reported_seq': 9833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:34.593583+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:34.593583+0000', 'last_peered': '2024-09-15T21:11:34.593583+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:34.593583+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9748, 'log_dups_size': 0, 'ondisk_log_size': 9748, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21985950, 'num_objects': 2685, 'num_object_clones': 0, 'num_object_copies': 10740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2685, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9747, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 427}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9904", 'reported_seq': 9979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:36.721355+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:36.721355+0000', 'last_peered': '2024-09-15T21:11:36.721355+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:36.721355+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9904, 'log_dups_size': 0, 'ondisk_log_size': 9904, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20903180, 'num_objects': 2552, 'num_object_clones': 0, 'num_object_copies': 10208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2552, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9902, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 218}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10002", 'reported_seq': 10064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.943656+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:37.943656+0000', 'last_peered': '2024-09-15T21:11:37.943656+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:37.943656+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21763340, 'num_objects': 2657, 'num_object_clones': 0, 'num_object_copies': 10628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10000, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9798", 'reported_seq': 9848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.959275+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:37.959275+0000', 'last_peered': '2024-09-15T21:11:37.959275+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:37.959275+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9798, 'log_dups_size': 0, 'ondisk_log_size': 9798, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21118976, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9798, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10057", 'reported_seq': 10133, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:36.660909+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:36.660909+0000', 'last_peered': '2024-09-15T21:11:36.660909+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:36.660909+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22238476, 'num_objects': 2715, 'num_object_clones': 0, 'num_object_copies': 10860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2715, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10055, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 349}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10152", 'reported_seq': 10227, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:36.820877+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:36.820877+0000', 'last_peered': '2024-09-15T21:11:36.820877+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:36.820877+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21370124, 'num_objects': 2609, 'num_object_clones': 0, 'num_object_copies': 10436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2609, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10150, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 135}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9851", 'reported_seq': 9915, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:38.010614+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:38.010614+0000', 'last_peered': '2024-09-15T21:11:38.010614+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:38.010614+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9851, 'log_dups_size': 0, 'ondisk_log_size': 9851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21301784, 'num_objects': 2601, 'num_object_clones': 0, 'num_object_copies': 10404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 269}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9922", 'reported_seq': 10009, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:34.482533+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:34.482533+0000', 'last_peered': '2024-09-15T21:11:34.482533+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:34.482533+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9922, 'log_dups_size': 0, 'ondisk_log_size': 9922, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21624076, 'num_objects': 2640, 'num_object_clones': 0, 'num_object_copies': 10560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2640, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9920, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 305}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9813", 'reported_seq': 9858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.917301+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:37.917301+0000', 'last_peered': '2024-09-15T21:11:37.917301+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:37.917301+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21412486, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 10456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9812, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9805", 'reported_seq': 11022, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:34.582315+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:34.582315+0000', 'last_peered': '2024-09-15T21:11:34.582315+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:34.582315+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9805, 'log_dups_size': 0, 'ondisk_log_size': 9805, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22084230, 'num_objects': 2696, 'num_object_clones': 0, 'num_object_copies': 10784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1721, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9804, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 567, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9850", 'reported_seq': 9912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.627131+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:37.627131+0000', 'last_peered': '2024-09-15T21:11:37.627131+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:37.627131+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21148940, 'num_objects': 2582, 'num_object_clones': 0, 'num_object_copies': 10328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9921", 'reported_seq': 10009, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:34.580229+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:34.580229+0000', 'last_peered': '2024-09-15T21:11:34.580229+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:34.580229+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9921, 'log_dups_size': 0, 'ondisk_log_size': 9921, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22607116, 'num_objects': 2760, 'num_object_clones': 0, 'num_object_copies': 11040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2760, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9919, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 479}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'9945", 'reported_seq': 10019, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:36.598329+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:36.598329+0000', 'last_peered': '2024-09-15T21:11:36.598329+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:36.598329+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9945, 'log_dups_size': 0, 'ondisk_log_size': 9945, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21329164, 'num_objects': 2604, 'num_object_clones': 0, 'num_object_copies': 10416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2604, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9943, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:38.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:39.040 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:39.040 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31875/168376 objects degraded (18.931%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":564,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":42094,"data_bytes":344793634,"bytes_used":2924146688,"bytes_avail":383622909952,"bytes_total":386547056640,"degraded_objects":31875,"degraded_total":168376,"degraded_ratio":0.18930845251104672,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":226},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:39.061 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:39.061 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:39.665 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:39.666 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:39.709 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-15T21:11:38.116272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344793634,"num_objects":42094,"num_object_clones":0,"num_object_copies":168376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142984,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158129,"ondisk_log_size":158129,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2855612,"kb_used_data":677544,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374631748,"statfs":{"total":386547056640,"available":383622909952,"internally_reserved":0,"allocated":693805056,"data_stored":693112648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,6,11,5,7,18,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22241280,"num_objects":-2715,"num_object_clones":0,"num_object_copies":-10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2715,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005233"},"pg_stats":[{"pgid":"1.f","version":"35'9755","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.052650+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:38.052650+0000","last_peered":"2024-09-15T21:11:38.052650+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:38.052650+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9932","reported_seq":14825,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:36.236919+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:36.236919+0000","last_peered":"2024-09-15T21:11:36.236919+0000","last_clean":"2024-09-15T21:11:36.236919+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:36.236919+0000","last_undegraded":"2024-09-15T21:11:36.236919+0000","last_fullsized":"2024-09-15T21:11:36.236919+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9932,"log_dups_size":0,"ondisk_log_size":9932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21446656,"num_objects":2618,"num_object_clones":0,"num_object_copies":10472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9748","reported_seq":9833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.593583+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:34.593583+0000","last_peered":"2024-09-15T21:11:34.593583+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:34.593583+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9748,"log_dups_size":0,"ondisk_log_size":9748,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21985950,"num_objects":2685,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9747,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":427}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9904","reported_seq":9979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.721355+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:36.721355+0000","last_peered":"2024-09-15T21:11:36.721355+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:36.721355+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9904,"log_dups_size":0,"ondisk_log_size":9904,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20903180,"num_objects":2552,"num_object_clones":0,"num_object_copies":10208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9902,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":218}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.943656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:37.943656+0000","last_peered":"2024-09-15T21:11:37.943656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:37.943656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9798","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.959275+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:37.959275+0000","last_peered":"2024-09-15T21:11:37.959275+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:37.959275+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10057","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.660909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:36.660909+0000","last_peered":"2024-09-15T21:11:36.660909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:36.660909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22238476,"num_objects":2715,"num_object_clones":0,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10055,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":349}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10152","reported_seq":10227,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.820877+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:36.820877+0000","last_peered":"2024-09-15T21:11:36.820877+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:36.820877+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21370124,"num_objects":2609,"num_object_clones":0,"num_object_copies":10436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10150,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":135}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9851","reported_seq":9915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.010614+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:38.010614+0000","last_peered":"2024-09-15T21:11:38.010614+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:38.010614+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21301784,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9922","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.482533+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:34.482533+0000","last_peered":"2024-09-15T21:11:34.482533+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:34.482533+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9922,"log_dups_size":0,"ondisk_log_size":9922,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21624076,"num_objects":2640,"num_object_clones":0,"num_object_copies":10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9920,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":305}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9774","reported_seq":14443,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:38.071344+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:38.071344+0000","last_peered":"2024-09-15T21:11:38.071344+0000","last_clean":"2024-09-15T21:11:38.071344+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:38.071344+0000","last_undegraded":"2024-09-15T21:11:38.071344+0000","last_fullsized":"2024-09-15T21:11:38.071344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21184512,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9813","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.917301+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:37.917301+0000","last_peered":"2024-09-15T21:11:37.917301+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:37.917301+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412486,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9805","reported_seq":11022,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.582315+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:34.582315+0000","last_peered":"2024-09-15T21:11:34.582315+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:34.582315+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9805,"log_dups_size":0,"ondisk_log_size":9805,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22084230,"num_objects":2696,"num_object_clones":0,"num_object_copies":10784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9804,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":567,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":975}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9850","reported_seq":9912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.627131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:37.627131+0000","last_peered":"2024-09-15T21:11:37.627131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:37.627131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9921","reported_seq":10009,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:34.580229+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:34.580229+0000","last_peered":"2024-09-15T21:11:34.580229+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:34.580229+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9921,"log_dups_size":0,"ondisk_log_size":9921,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22607116,"num_objects":2760,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9919,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":479}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'9945","reported_seq":10019,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:36.598329+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:36.598329+0000","last_peered":"2024-09-15T21:11:36.598329+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:36.598329+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9945,"log_dups_size":0,"ondisk_log_size":9945,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21329164,"num_objects":2604,"num_object_clones":0,"num_object_copies":10416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9943,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":344793634,"num_objects":42094,"num_object_clones":0,"num_object_copies":168376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142984,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":690720768,"data_stored":690720768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158129,"ondisk_log_size":158129,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741288,"kb_used_data":168152,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93630552,"statfs":{"total":96636764160,"available":95877685248,"internally_reserved":0,"allocated":172187648,"data_stored":172014546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,1,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747676,"kb_used_data":170872,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93624164,"statfs":{"total":96636764160,"available":95871143936,"internally_reserved":0,"allocated":174972928,"data_stored":174799826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,3,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051676,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598292,"kb_used_data":169604,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93773548,"statfs":{"total":96636764160,"available":96024113152,"internally_reserved":0,"allocated":173674496,"data_stored":173501394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247914,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":768356,"kb_used_data":168916,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93603484,"statfs":{"total":96636764160,"available":95849967616,"internally_reserved":0,"allocated":172969984,"data_stored":172796882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,3,0,2,4,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172290048,"data_stored":172290048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172875776,"data_stored":172875776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174215168,"data_stored":174215168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171339776,"data_stored":171339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:42.711 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:43.343 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:43.346 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:43.367 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-15T21:11:42.117563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340189730,"num_objects":41532,"num_object_clones":0,"num_object_copies":166128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143546,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158491,"ondisk_log_size":158491,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839564,"kb_used_data":665676,"kb_used_omap":14,"kb_used_meta":2173809,"kb_avail":374647796,"statfs":{"total":386547056640,"available":383639343104,"internally_reserved":0,"allocated":681652224,"data_stored":680959816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2225980978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,7,14,10,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19308544,"num_objects":-2357,"num_object_clones":0,"num_object_copies":-9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2357,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004662"},"pg_stats":[{"pgid":"1.f","version":"35'9755","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.052650+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:38.052650+0000","last_peered":"2024-09-15T21:11:38.052650+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:38.052650+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9995","reported_seq":14888,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:41.190526+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:41.190526+0000","last_peered":"2024-09-15T21:11:41.190526+0000","last_clean":"2024-09-15T21:11:41.190526+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:41.190526+0000","last_undegraded":"2024-09-15T21:11:41.190526+0000","last_fullsized":"2024-09-15T21:11:41.190526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20930560,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9813","reported_seq":9897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.392776+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:39.392776+0000","last_peered":"2024-09-15T21:11:39.392776+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:39.392776+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453470,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9812,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9967","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.933722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:41.933722+0000","last_peered":"2024-09-15T21:11:41.933722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:41.933722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.943656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:37.943656+0000","last_peered":"2024-09-15T21:11:37.943656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:37.943656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9798","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.959275+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:37.959275+0000","last_peered":"2024-09-15T21:11:37.959275+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:37.959275+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10128","reported_seq":10204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.916991+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:41.916991+0000","last_peered":"2024-09-15T21:11:41.916991+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:41.916991+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656844,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10210","reported_seq":10285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.852540+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:41.852540+0000","last_peered":"2024-09-15T21:11:41.852540+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:41.852540+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20894988,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9851","reported_seq":9915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.010614+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:38.010614+0000","last_peered":"2024-09-15T21:11:38.010614+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:38.010614+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21301784,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9979","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.488964+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:39.488964+0000","last_peered":"2024-09-15T21:11:39.488964+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:39.488964+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157132,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":248}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9774","reported_seq":14443,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:38.071344+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:38.071344+0000","last_peered":"2024-09-15T21:11:38.071344+0000","last_clean":"2024-09-15T21:11:38.071344+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:38.071344+0000","last_undegraded":"2024-09-15T21:11:38.071344+0000","last_fullsized":"2024-09-15T21:11:38.071344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21184512,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9813","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.917301+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:37.917301+0000","last_peered":"2024-09-15T21:11:37.917301+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:37.917301+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412486,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9871","reported_seq":11191,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.680578+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:39.680578+0000","last_peered":"2024-09-15T21:11:39.680578+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:39.680578+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543558,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9850","reported_seq":9912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.627131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:37.627131+0000","last_peered":"2024-09-15T21:11:37.627131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:37.627131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9981","reported_seq":10070,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.683002+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:39.683002+0000","last_peered":"2024-09-15T21:11:39.683002+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:39.683002+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10004","reported_seq":10078,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.867859+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:41.867859+0000","last_peered":"2024-09-15T21:11:41.867859+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:41.867859+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845836,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":340189730,"num_objects":41532,"num_object_clones":0,"num_object_copies":166128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143546,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678408192,"data_stored":678408192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158491,"ondisk_log_size":158491,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741288,"kb_used_data":168152,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93630552,"statfs":{"total":96636764160,"available":95877685248,"internally_reserved":0,"allocated":172187648,"data_stored":172014546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,1,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747792,"kb_used_data":166848,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93624048,"statfs":{"total":96636764160,"available":95871025152,"internally_reserved":0,"allocated":170852352,"data_stored":170679250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598536,"kb_used_data":165688,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93773304,"statfs":{"total":96636764160,"available":96023863296,"internally_reserved":0,"allocated":169664512,"data_stored":169491410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,6,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751948,"kb_used_data":164988,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93619892,"statfs":{"total":96636764160,"available":95866769408,"internally_reserved":0,"allocated":168947712,"data_stored":168774610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168161280,"data_stored":168161280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168812544,"data_stored":168812544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170094592,"data_stored":170094592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171339776,"data_stored":171339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:43.368 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:43.947 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:43.949 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:43.967 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-09-15T21:11:42.117563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":340189730,"num_objects":41532,"num_object_clones":0,"num_object_copies":166128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143546,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158491,"ondisk_log_size":158491,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839564,"kb_used_data":665676,"kb_used_omap":14,"kb_used_meta":2173809,"kb_avail":374647796,"statfs":{"total":386547056640,"available":383639343104,"internally_reserved":0,"allocated":681652224,"data_stored":680959816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2225980978},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,9,7,14,10,12,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19308544,"num_objects":-2357,"num_object_clones":0,"num_object_copies":-9428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2357,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004662"},"pg_stats":[{"pgid":"1.f","version":"35'9755","reported_seq":9805,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.052650+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:38.052650+0000","last_peered":"2024-09-15T21:11:38.052650+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:38.052650+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9755,"log_dups_size":0,"ondisk_log_size":9755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21274624,"num_objects":2597,"num_object_clones":0,"num_object_copies":10388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9755,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":321}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9995","reported_seq":14888,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:41.190526+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:41.190526+0000","last_peered":"2024-09-15T21:11:41.190526+0000","last_clean":"2024-09-15T21:11:41.190526+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:41.190526+0000","last_undegraded":"2024-09-15T21:11:41.190526+0000","last_fullsized":"2024-09-15T21:11:41.190526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20930560,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9813","reported_seq":9897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.392776+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:39.392776+0000","last_peered":"2024-09-15T21:11:39.392776+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:39.392776+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453470,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9812,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9967","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.933722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:41.933722+0000","last_peered":"2024-09-15T21:11:41.933722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:41.933722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10002","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.943656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:37.943656+0000","last_peered":"2024-09-15T21:11:37.943656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:37.943656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21763340,"num_objects":2657,"num_object_clones":0,"num_object_copies":10628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10000,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":346}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9798","reported_seq":9848,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.959275+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:37.959275+0000","last_peered":"2024-09-15T21:11:37.959275+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:37.959275+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9798,"log_dups_size":0,"ondisk_log_size":9798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21118976,"num_objects":2578,"num_object_clones":0,"num_object_copies":10312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9798,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":262}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10128","reported_seq":10204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.916991+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:41.916991+0000","last_peered":"2024-09-15T21:11:41.916991+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:41.916991+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656844,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10210","reported_seq":10285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.852540+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:41.852540+0000","last_peered":"2024-09-15T21:11:41.852540+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:41.852540+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20894988,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9851","reported_seq":9915,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:38.010614+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:38.010614+0000","last_peered":"2024-09-15T21:11:38.010614+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:38.010614+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9851,"log_dups_size":0,"ondisk_log_size":9851,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21301784,"num_objects":2601,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9979","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.488964+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:39.488964+0000","last_peered":"2024-09-15T21:11:39.488964+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:39.488964+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157132,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":248}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9774","reported_seq":14443,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:38.071344+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:38.071344+0000","last_peered":"2024-09-15T21:11:38.071344+0000","last_clean":"2024-09-15T21:11:38.071344+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:38.071344+0000","last_undegraded":"2024-09-15T21:11:38.071344+0000","last_fullsized":"2024-09-15T21:11:38.071344+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9774,"log_dups_size":0,"ondisk_log_size":9774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21184512,"num_objects":2586,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3885,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9813","reported_seq":9858,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.917301+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:37.917301+0000","last_peered":"2024-09-15T21:11:37.917301+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:37.917301+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21412486,"num_objects":2614,"num_object_clones":0,"num_object_copies":10456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9871","reported_seq":11191,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.680578+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:39.680578+0000","last_peered":"2024-09-15T21:11:39.680578+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:39.680578+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543558,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9850","reported_seq":9912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:37.627131+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:37.627131+0000","last_peered":"2024-09-15T21:11:37.627131+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:37.627131+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9850,"log_dups_size":0,"ondisk_log_size":9850,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9981","reported_seq":10070,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.683002+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:39.683002+0000","last_peered":"2024-09-15T21:11:39.683002+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:39.683002+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10004","reported_seq":10078,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.867859+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:41.867859+0000","last_peered":"2024-09-15T21:11:41.867859+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:41.867859+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845836,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":340189730,"num_objects":41532,"num_object_clones":0,"num_object_copies":166128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41532,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143546,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":678408192,"data_stored":678408192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158491,"ondisk_log_size":158491,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741288,"kb_used_data":168152,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93630552,"statfs":{"total":96636764160,"available":95877685248,"internally_reserved":0,"allocated":172187648,"data_stored":172014546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,5,3,1,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747792,"kb_used_data":166848,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93624048,"statfs":{"total":96636764160,"available":95871025152,"internally_reserved":0,"allocated":170852352,"data_stored":170679250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598536,"kb_used_data":165688,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93773304,"statfs":{"total":96636764160,"available":96023863296,"internally_reserved":0,"allocated":169664512,"data_stored":169491410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,6,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751948,"kb_used_data":164988,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93619892,"statfs":{"total":96636764160,"available":95866769408,"internally_reserved":0,"allocated":168947712,"data_stored":168774610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168161280,"data_stored":168161280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168812544,"data_stored":168812544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170094592,"data_stored":170094592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":171339776,"data_stored":171339776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:43.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:43.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9755", 'reported_seq': 9805, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:38.052650+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:38.052650+0000', 'last_peered': '2024-09-15T21:11:38.052650+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:38.052650+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9755, 'log_dups_size': 0, 'ondisk_log_size': 9755, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21274624, 'num_objects': 2597, 'num_object_clones': 0, 'num_object_copies': 10388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2597, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9755, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 321}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:43.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9813", 'reported_seq': 9897, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:39.392776+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:39.392776+0000', 'last_peered': '2024-09-15T21:11:39.392776+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:39.392776+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21453470, 'num_objects': 2620, 'num_object_clones': 0, 'num_object_copies': 10480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2620, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9812, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 362}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'9967", 'reported_seq': 10042, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:41.933722+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:41.933722+0000', 'last_peered': '2024-09-15T21:11:41.933722+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:41.933722+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9967, 'log_dups_size': 0, 'ondisk_log_size': 9967, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20387084, 'num_objects': 2489, 'num_object_clones': 0, 'num_object_copies': 9956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9965, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 155}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10002", 'reported_seq': 10064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.943656+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:37.943656+0000', 'last_peered': '2024-09-15T21:11:37.943656+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:37.943656+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21763340, 'num_objects': 2657, 'num_object_clones': 0, 'num_object_copies': 10628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2657, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10000, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 346}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9798", 'reported_seq': 9848, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.959275+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:37.959275+0000', 'last_peered': '2024-09-15T21:11:37.959275+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:37.959275+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9798, 'log_dups_size': 0, 'ondisk_log_size': 9798, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21118976, 'num_objects': 2578, 'num_object_clones': 0, 'num_object_copies': 10312, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2578, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9798, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 262}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10128", 'reported_seq': 10204, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:41.916991+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:41.916991+0000', 'last_peered': '2024-09-15T21:11:41.916991+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:41.916991+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21656844, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10126, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 278}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10210", 'reported_seq': 10285, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:41.852540+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:41.852540+0000', 'last_peered': '2024-09-15T21:11:41.852540+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:41.852540+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20894988, 'num_objects': 2551, 'num_object_clones': 0, 'num_object_copies': 10204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2551, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10208, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 77}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:43.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9851", 'reported_seq': 9915, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:38.010614+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:38.010614+0000', 'last_peered': '2024-09-15T21:11:38.010614+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:38.010614+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9851, 'log_dups_size': 0, 'ondisk_log_size': 9851, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21301784, 'num_objects': 2601, 'num_object_clones': 0, 'num_object_copies': 10404, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2601, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 269}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'9979", 'reported_seq': 10066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:39.488964+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:39.488964+0000', 'last_peered': '2024-09-15T21:11:39.488964+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:39.488964+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9979, 'log_dups_size': 0, 'ondisk_log_size': 9979, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21157132, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9977, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 248}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9813", 'reported_seq': 9858, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.917301+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:37.917301+0000', 'last_peered': '2024-09-15T21:11:37.917301+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:37.917301+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9813, 'log_dups_size': 0, 'ondisk_log_size': 9813, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21412486, 'num_objects': 2614, 'num_object_clones': 0, 'num_object_copies': 10456, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2614, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9812, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9871", 'reported_seq': 11191, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:39.680578+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:39.680578+0000', 'last_peered': '2024-09-15T21:11:39.680578+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:39.680578+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9871, 'log_dups_size': 0, 'ondisk_log_size': 9871, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21543558, 'num_objects': 2630, 'num_object_clones': 0, 'num_object_copies': 10520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1670, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2630, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9870, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 618, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 960}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9850", 'reported_seq': 9912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:37.627131+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:37.627131+0000', 'last_peered': '2024-09-15T21:11:37.627131+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:37.627131+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9850, 'log_dups_size': 0, 'ondisk_log_size': 9850, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21148940, 'num_objects': 2582, 'num_object_clones': 0, 'num_object_copies': 10328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9848, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 303}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'9981", 'reported_seq': 10070, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:39.683002+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:39.683002+0000', 'last_peered': '2024-09-15T21:11:39.683002+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:39.683002+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9981, 'log_dups_size': 0, 'ondisk_log_size': 9981, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 22115596, 'num_objects': 2700, 'num_object_clones': 0, 'num_object_copies': 10800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2700, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9979, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 419}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10004", 'reported_seq': 10078, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:41.867859+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:41.867859+0000', 'last_peered': '2024-09-15T21:11:41.867859+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:41.867859+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20845836, 'num_objects': 2545, 'num_object_clones': 0, 'num_object_copies': 10180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2545, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10002, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 230}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:43.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:44.673 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:44.673 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31824/164324 objects degraded (19.367%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":570,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":41081,"data_bytes":336495138,"bytes_used":2907807744,"bytes_avail":383639248896,"bytes_total":386547056640,"degraded_objects":31824,"degraded_total":164324,"degraded_ratio":0.19366617170954942,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":207},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:44.695 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:44.695 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:45.272 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:45.273 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:45.298 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-15T21:11:44.121109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":336495138,"num_objects":41081,"num_object_clones":0,"num_object_copies":164324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41081,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143997,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158942,"ondisk_log_size":158942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2839656,"kb_used_data":661608,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374647704,"statfs":{"total":386547056640,"available":383639248896,"internally_reserved":0,"allocated":677486592,"data_stored":676794184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,20,4,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20406272,"num_objects":-2491,"num_object_clones":0,"num_object_copies":-9964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2491,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007373"},"pg_stats":[{"pgid":"1.f","version":"35'9812","reported_seq":9863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.948376+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:42.948376+0000","last_peered":"2024-09-15T21:11:42.948376+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:42.948376+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'9995","reported_seq":14888,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:41.190526+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:41.190526+0000","last_peered":"2024-09-15T21:11:41.190526+0000","last_clean":"2024-09-15T21:11:41.190526+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:41.190526+0000","last_undegraded":"2024-09-15T21:11:41.190526+0000","last_fullsized":"2024-09-15T21:11:41.190526+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":9995,"log_dups_size":0,"ondisk_log_size":9995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20930560,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":661,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9813","reported_seq":9897,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.392776+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:39.392776+0000","last_peered":"2024-09-15T21:11:39.392776+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:39.392776+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9813,"log_dups_size":0,"ondisk_log_size":9813,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21453470,"num_objects":2620,"num_object_clones":0,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9812,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":362}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'9967","reported_seq":10042,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.933722+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:41.933722+0000","last_peered":"2024-09-15T21:11:41.933722+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:41.933722+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9967,"log_dups_size":0,"ondisk_log_size":9967,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9965,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":155}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10070","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.820356+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:42.820356+0000","last_peered":"2024-09-15T21:11:42.820356+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:42.820356+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206284,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10068,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9864","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.913010+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:42.913010+0000","last_peered":"2024-09-15T21:11:42.913010+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:42.913010+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10128","reported_seq":10204,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.916991+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:41.916991+0000","last_peered":"2024-09-15T21:11:41.916991+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:41.916991+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656844,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10126,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":278}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10210","reported_seq":10285,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.852540+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:41.852540+0000","last_peered":"2024-09-15T21:11:41.852540+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:41.852540+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20894988,"num_objects":2551,"num_object_clones":0,"num_object_copies":10204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":77}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9924","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.805779+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:42.805779+0000","last_peered":"2024-09-15T21:11:42.805779+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:42.805779+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703768,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'9979","reported_seq":10066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.488964+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:39.488964+0000","last_peered":"2024-09-15T21:11:39.488964+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:39.488964+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9979,"log_dups_size":0,"ondisk_log_size":9979,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21157132,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9977,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":248}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9842","reported_seq":14511,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:43.015000+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:43.015000+0000","last_peered":"2024-09-15T21:11:43.015000+0000","last_clean":"2024-09-15T21:11:43.015000+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:43.015000+0000","last_undegraded":"2024-09-15T21:11:43.015000+0000","last_fullsized":"2024-09-15T21:11:43.015000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.941330+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:42.941330+0000","last_peered":"2024-09-15T21:11:42.941330+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:42.941330+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20863622,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9879,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9871","reported_seq":11191,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.680578+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:39.680578+0000","last_peered":"2024-09-15T21:11:39.680578+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:39.680578+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9871,"log_dups_size":0,"ondisk_log_size":9871,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21543558,"num_objects":2630,"num_object_clones":0,"num_object_copies":10520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1670,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":960}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9902","reported_seq":9964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:43.013104+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:43.013104+0000","last_peered":"2024-09-15T21:11:43.013104+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:43.013104+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722956,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":251}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'9981","reported_seq":10070,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:39.683002+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:39.683002+0000","last_peered":"2024-09-15T21:11:39.683002+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:39.683002+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9981,"log_dups_size":0,"ondisk_log_size":9981,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22115596,"num_objects":2700,"num_object_clones":0,"num_object_copies":10800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9979,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":419}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10004","reported_seq":10078,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:41.867859+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:41.867859+0000","last_peered":"2024-09-15T21:11:41.867859+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:41.867859+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20845836,"num_objects":2545,"num_object_clones":0,"num_object_copies":10180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10002,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":230}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":336495138,"num_objects":41081,"num_object_clones":0,"num_object_copies":164324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41081,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":143997,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":674308096,"data_stored":674308096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158942,"ondisk_log_size":158942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741380,"kb_used_data":164084,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93630460,"statfs":{"total":96636764160,"available":95877591040,"internally_reserved":0,"allocated":168022016,"data_stored":167848914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747792,"kb_used_data":166848,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93624048,"statfs":{"total":96636764160,"available":95871025152,"internally_reserved":0,"allocated":170852352,"data_stored":170679250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051677,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598536,"kb_used_data":165688,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93773304,"statfs":{"total":96636764160,"available":96023863296,"internally_reserved":0,"allocated":169664512,"data_stored":169491410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,6,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247915,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751948,"kb_used_data":164988,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93619892,"statfs":{"total":96636764160,"available":95866769408,"internally_reserved":0,"allocated":168947712,"data_stored":168774610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,3,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":168161280,"data_stored":168161280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":168812544,"data_stored":168812544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":170094592,"data_stored":170094592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167239680,"data_stored":167239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:48.300 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:48.881 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:48.882 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:48.910 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-15T21:11:48.122643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":332169762,"num_objects":40553,"num_object_clones":0,"num_object_copies":162212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144525,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159470,"ondisk_log_size":159470,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2840300,"kb_used_data":649772,"kb_used_omap":14,"kb_used_meta":2190449,"kb_avail":374647060,"statfs":{"total":386547056640,"available":383638589440,"internally_reserved":0,"allocated":665366528,"data_stored":664674120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2243020338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,7,2,14,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18800640,"num_objects":-2295,"num_object_clones":0,"num_object_copies":-9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007260"},"pg_stats":[{"pgid":"1.f","version":"35'9812","reported_seq":9863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.948376+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:42.948376+0000","last_peered":"2024-09-15T21:11:42.948376+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:42.948376+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10063","reported_seq":14954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:46.350869+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:46.350869+0000","last_peered":"2024-09-15T21:11:46.350869+0000","last_clean":"2024-09-15T21:11:46.350869+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:46.350869+0000","last_undegraded":"2024-09-15T21:11:46.350869+0000","last_fullsized":"2024-09-15T21:11:46.350869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20373504,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9878","reported_seq":9963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.687170+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:44.687170+0000","last_peered":"2024-09-15T21:11:44.687170+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:44.687170+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20920990,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9877,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10023","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.907113+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:46.907113+0000","last_peered":"2024-09-15T21:11:46.907113+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:46.907113+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19928332,"num_objects":2433,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10070","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.820356+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:42.820356+0000","last_peered":"2024-09-15T21:11:42.820356+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:42.820356+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206284,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10068,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9864","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.913010+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:42.913010+0000","last_peered":"2024-09-15T21:11:42.913010+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:42.913010+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10187","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.939642+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:46.939642+0000","last_peered":"2024-09-15T21:11:46.939642+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:46.939642+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173516,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":219}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10272","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.920002+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:46.920002+0000","last_peered":"2024-09-15T21:11:46.920002+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:46.920002+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9924","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.805779+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:42.805779+0000","last_peered":"2024-09-15T21:11:42.805779+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:42.805779+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703768,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10039","reported_seq":10126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.604474+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:44.604474+0000","last_peered":"2024-09-15T21:11:44.604474+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:44.604474+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20665612,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":188}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9842","reported_seq":14511,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:43.015000+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:43.015000+0000","last_peered":"2024-09-15T21:11:43.015000+0000","last_clean":"2024-09-15T21:11:43.015000+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:43.015000+0000","last_undegraded":"2024-09-15T21:11:43.015000+0000","last_fullsized":"2024-09-15T21:11:43.015000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.941330+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:42.941330+0000","last_peered":"2024-09-15T21:11:42.941330+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:42.941330+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20863622,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9879,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9918","reported_seq":11339,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.748467+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:44.748467+0000","last_peered":"2024-09-15T21:11:44.748467+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:44.748467+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158534,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9902","reported_seq":9964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:43.013104+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:43.013104+0000","last_peered":"2024-09-15T21:11:43.013104+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:43.013104+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722956,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":251}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10037","reported_seq":10126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.686478+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:44.686478+0000","last_peered":"2024-09-15T21:11:44.686478+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:44.686478+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656844,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":363}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10059","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.792335+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:46.792335+0000","last_peered":"2024-09-15T21:11:46.792335+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:46.792335+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20395276,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":332169762,"num_objects":40553,"num_object_clones":0,"num_object_copies":162212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144525,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662220800,"data_stored":662220800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159470,"ondisk_log_size":159470,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741380,"kb_used_data":164084,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93630460,"statfs":{"total":96636764160,"available":95877591040,"internally_reserved":0,"allocated":168022016,"data_stored":167848914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747940,"kb_used_data":162836,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":93623900,"statfs":{"total":96636764160,"available":95870873600,"internally_reserved":0,"allocated":166744064,"data_stored":166570962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598792,"kb_used_data":161784,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93773048,"statfs":{"total":96636764160,"available":96023601152,"internally_reserved":0,"allocated":165666816,"data_stored":165493714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752188,"kb_used_data":161068,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93619652,"statfs":{"total":96636764160,"available":95866523648,"internally_reserved":0,"allocated":164933632,"data_stored":164760530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164245504,"data_stored":164245504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164765696,"data_stored":164765696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165969920,"data_stored":165969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167239680,"data_stored":167239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:48.911 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:49.507 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:49.507 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:49.540 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-15T21:11:48.122643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":332169762,"num_objects":40553,"num_object_clones":0,"num_object_copies":162212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144525,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159470,"ondisk_log_size":159470,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2840300,"kb_used_data":649772,"kb_used_omap":14,"kb_used_meta":2190449,"kb_avail":374647060,"statfs":{"total":386547056640,"available":383638589440,"internally_reserved":0,"allocated":665366528,"data_stored":664674120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2243020338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,7,2,14,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18800640,"num_objects":-2295,"num_object_clones":0,"num_object_copies":-9180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007260"},"pg_stats":[{"pgid":"1.f","version":"35'9812","reported_seq":9863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.948376+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:42.948376+0000","last_peered":"2024-09-15T21:11:42.948376+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:42.948376+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9812,"log_dups_size":0,"ondisk_log_size":9812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20807680,"num_objects":2540,"num_object_clones":0,"num_object_copies":10160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9812,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10063","reported_seq":14954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:46.350869+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:46.350869+0000","last_peered":"2024-09-15T21:11:46.350869+0000","last_clean":"2024-09-15T21:11:46.350869+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:46.350869+0000","last_undegraded":"2024-09-15T21:11:46.350869+0000","last_fullsized":"2024-09-15T21:11:46.350869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20373504,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9878","reported_seq":9963,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.687170+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:44.687170+0000","last_peered":"2024-09-15T21:11:44.687170+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:44.687170+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9878,"log_dups_size":0,"ondisk_log_size":9878,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20920990,"num_objects":2555,"num_object_clones":0,"num_object_copies":10220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9877,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":297}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10023","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.907113+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:46.907113+0000","last_peered":"2024-09-15T21:11:46.907113+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:46.907113+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19928332,"num_objects":2433,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10070","reported_seq":10132,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.820356+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:42.820356+0000","last_peered":"2024-09-15T21:11:42.820356+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:42.820356+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21206284,"num_objects":2589,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10068,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":278}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9864","reported_seq":9914,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.913010+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:42.913010+0000","last_peered":"2024-09-15T21:11:42.913010+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:42.913010+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9864,"log_dups_size":0,"ondisk_log_size":9864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":2512,"num_object_clones":0,"num_object_copies":10048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9864,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10187","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.939642+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:46.939642+0000","last_peered":"2024-09-15T21:11:46.939642+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:46.939642+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173516,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":219}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10272","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.920002+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:46.920002+0000","last_peered":"2024-09-15T21:11:46.920002+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:46.920002+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9924","reported_seq":9990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.805779+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:42.805779+0000","last_peered":"2024-09-15T21:11:42.805779+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:42.805779+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20703768,"num_objects":2528,"num_object_clones":0,"num_object_copies":10112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9921,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10039","reported_seq":10126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.604474+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:44.604474+0000","last_peered":"2024-09-15T21:11:44.604474+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:44.604474+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20665612,"num_objects":2523,"num_object_clones":0,"num_object_copies":10092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10037,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":188}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9842","reported_seq":14511,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:43.015000+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:43.015000+0000","last_peered":"2024-09-15T21:11:43.015000+0000","last_clean":"2024-09-15T21:11:43.015000+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:43.015000+0000","last_undegraded":"2024-09-15T21:11:43.015000+0000","last_fullsized":"2024-09-15T21:11:43.015000+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9842,"log_dups_size":0,"ondisk_log_size":9842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20627456,"num_objects":2518,"num_object_clones":0,"num_object_copies":10072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3953,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9880","reported_seq":9925,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:42.941330+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:42.941330+0000","last_peered":"2024-09-15T21:11:42.941330+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:42.941330+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9880,"log_dups_size":0,"ondisk_log_size":9880,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20863622,"num_objects":2547,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9879,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":291}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9918","reported_seq":11339,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.748467+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:44.748467+0000","last_peered":"2024-09-15T21:11:44.748467+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:44.748467+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9918,"log_dups_size":0,"ondisk_log_size":9918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21158534,"num_objects":2583,"num_object_clones":0,"num_object_copies":10332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9917,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":963}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9902","reported_seq":9964,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:43.013104+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:43.013104+0000","last_peered":"2024-09-15T21:11:43.013104+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:43.013104+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9902,"log_dups_size":0,"ondisk_log_size":9902,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20722956,"num_objects":2530,"num_object_clones":0,"num_object_copies":10120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":251}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10037","reported_seq":10126,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:44.686478+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:44.686478+0000","last_peered":"2024-09-15T21:11:44.686478+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:44.686478+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21656844,"num_objects":2644,"num_object_clones":0,"num_object_copies":10576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":363}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10059","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.792335+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:46.792335+0000","last_peered":"2024-09-15T21:11:46.792335+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:46.792335+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20395276,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":332169762,"num_objects":40553,"num_object_clones":0,"num_object_copies":162212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31774,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144525,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":689,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":662220800,"data_stored":662220800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159470,"ondisk_log_size":159470,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741380,"kb_used_data":164084,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93630460,"statfs":{"total":96636764160,"available":95877591040,"internally_reserved":0,"allocated":168022016,"data_stored":167848914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,7,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":747940,"kb_used_data":162836,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":93623900,"statfs":{"total":96636764160,"available":95870873600,"internally_reserved":0,"allocated":166744064,"data_stored":166570962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598792,"kb_used_data":161784,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93773048,"statfs":{"total":96636764160,"available":96023601152,"internally_reserved":0,"allocated":165666816,"data_stored":165493714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752188,"kb_used_data":161068,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93619652,"statfs":{"total":96636764160,"available":95866523648,"internally_reserved":0,"allocated":164933632,"data_stored":164760530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164245504,"data_stored":164245504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164765696,"data_stored":164765696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165969920,"data_stored":165969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167239680,"data_stored":167239680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:49.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:49.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9812", 'reported_seq': 9863, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:42.948376+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:42.948376+0000', 'last_peered': '2024-09-15T21:11:42.948376+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:42.948376+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9812, 'log_dups_size': 0, 'ondisk_log_size': 9812, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20807680, 'num_objects': 2540, 'num_object_clones': 0, 'num_object_copies': 10160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9812, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 264}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9878", 'reported_seq': 9963, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:44.687170+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:44.687170+0000', 'last_peered': '2024-09-15T21:11:44.687170+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:44.687170+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9878, 'log_dups_size': 0, 'ondisk_log_size': 9878, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20920990, 'num_objects': 2555, 'num_object_clones': 0, 'num_object_copies': 10220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2555, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9877, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 297}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10023", 'reported_seq': 10095, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:46.907113+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:46.907113+0000', 'last_peered': '2024-09-15T21:11:46.907113+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:46.907113+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19928332, 'num_objects': 2433, 'num_object_clones': 0, 'num_object_copies': 9732, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2433, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10021, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 99}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10070", 'reported_seq': 10132, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:42.820356+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:42.820356+0000', 'last_peered': '2024-09-15T21:11:42.820356+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:42.820356+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21206284, 'num_objects': 2589, 'num_object_clones': 0, 'num_object_copies': 10356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10068, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 278}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9864", 'reported_seq': 9914, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:42.913010+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:42.913010+0000', 'last_peered': '2024-09-15T21:11:42.913010+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:42.913010+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9864, 'log_dups_size': 0, 'ondisk_log_size': 9864, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20578304, 'num_objects': 2512, 'num_object_clones': 0, 'num_object_copies': 10048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2512, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9864, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:49.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10187", 'reported_seq': 10263, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:46.939642+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:46.939642+0000', 'last_peered': '2024-09-15T21:11:46.939642+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:46.939642+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21173516, 'num_objects': 2585, 'num_object_clones': 0, 'num_object_copies': 10340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2585, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10185, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 219}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10272", 'reported_seq': 10347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:46.920002+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:46.920002+0000', 'last_peered': '2024-09-15T21:11:46.920002+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:46.920002+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20387084, 'num_objects': 2489, 'num_object_clones': 0, 'num_object_copies': 9956, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2489, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10270, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 15}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'9924", 'reported_seq': 9990, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:42.805779+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:42.805779+0000', 'last_peered': '2024-09-15T21:11:42.805779+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:42.805779+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9924, 'log_dups_size': 0, 'ondisk_log_size': 9924, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20703768, 'num_objects': 2528, 'num_object_clones': 0, 'num_object_copies': 10112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9921, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10039", 'reported_seq': 10126, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:44.604474+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:44.604474+0000', 'last_peered': '2024-09-15T21:11:44.604474+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:44.604474+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20665612, 'num_objects': 2523, 'num_object_clones': 0, 'num_object_copies': 10092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2523, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10037, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 188}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'9880", 'reported_seq': 9925, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:42.941330+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:42.941330+0000', 'last_peered': '2024-09-15T21:11:42.941330+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:42.941330+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9880, 'log_dups_size': 0, 'ondisk_log_size': 9880, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20863622, 'num_objects': 2547, 'num_object_clones': 0, 'num_object_copies': 10188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2547, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9879, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 291}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:49.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9918", 'reported_seq': 11339, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:44.748467+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:44.748467+0000', 'last_peered': '2024-09-15T21:11:44.748467+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:44.748467+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9918, 'log_dups_size': 0, 'ondisk_log_size': 9918, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21158534, 'num_objects': 2583, 'num_object_clones': 0, 'num_object_copies': 10332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1620, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9917, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 668, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 963}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'9902", 'reported_seq': 9964, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:43.013104+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:43.013104+0000', 'last_peered': '2024-09-15T21:11:43.013104+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:43.013104+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9902, 'log_dups_size': 0, 'ondisk_log_size': 9902, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20722956, 'num_objects': 2530, 'num_object_clones': 0, 'num_object_copies': 10120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9900, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 251}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10037", 'reported_seq': 10126, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:44.686478+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:44.686478+0000', 'last_peered': '2024-09-15T21:11:44.686478+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:44.686478+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21656844, 'num_objects': 2644, 'num_object_clones': 0, 'num_object_copies': 10576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10035, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 363}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:49.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10059", 'reported_seq': 10133, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:46.792335+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:46.792335+0000', 'last_peered': '2024-09-15T21:11:46.792335+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:46.792335+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20395276, 'num_objects': 2490, 'num_object_clones': 0, 'num_object_copies': 9960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10057, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:49.544 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:50.321 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:50.321 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31774/162212 objects degraded (19.588%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":576,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":40553,"data_bytes":332169762,"bytes_used":2908467200,"bytes_avail":383638589440,"bytes_total":386547056640,"degraded_objects":31774,"degraded_total":162212,"degraded_ratio":0.19587946637733336,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":191},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:50.347 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:50.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:51.004 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:51.005 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:51.027 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-09-15T21:11:50.123314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":326345250,"num_objects":39842,"num_object_clones":0,"num_object_copies":159368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145236,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159981,"ondisk_log_size":159981,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2824088,"kb_used_data":641868,"kb_used_omap":14,"kb_used_meta":2182129,"kb_avail":374663272,"statfs":{"total":386547056640,"available":383655190528,"internally_reserved":0,"allocated":657272832,"data_stored":656580424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2234500658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,5,7,3,18,8,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18448384,"num_objects":-2252,"num_object_clones":0,"num_object_copies":-9008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2252,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007041"},"pg_stats":[{"pgid":"1.f","version":"35'9872","reported_seq":9923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:47.745317+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:47.745317+0000","last_peered":"2024-09-15T21:11:47.745317+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:47.745317+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9872,"log_dups_size":0,"ondisk_log_size":9872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20316160,"num_objects":2480,"num_object_clones":0,"num_object_copies":9920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9872,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":204}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10063","reported_seq":14954,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:46.350869+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:46.350869+0000","last_peered":"2024-09-15T21:11:46.350869+0000","last_clean":"2024-09-15T21:11:46.350869+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:46.350869+0000","last_undegraded":"2024-09-15T21:11:46.350869+0000","last_fullsized":"2024-09-15T21:11:46.350869+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20373504,"num_objects":2487,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9940","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.667300+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:49.667300+0000","last_peered":"2024-09-15T21:11:49.667300+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:49.667300+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20413086,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9939,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10023","reported_seq":10095,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.907113+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:46.907113+0000","last_peered":"2024-09-15T21:11:46.907113+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:46.907113+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19928332,"num_objects":2433,"num_object_clones":0,"num_object_copies":9732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10021,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10140","reported_seq":10202,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:48.120531+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:48.120531+0000","last_peered":"2024-09-15T21:11:48.120531+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:48.120531+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20632844,"num_objects":2519,"num_object_clones":0,"num_object_copies":10076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10138,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":208}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9924","reported_seq":9974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:48.029856+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:48.029856+0000","last_peered":"2024-09-15T21:11:48.029856+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:48.029856+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9924,"log_dups_size":0,"ondisk_log_size":9924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20086784,"num_objects":2452,"num_object_clones":0,"num_object_copies":9808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9924,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10187","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.939642+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:46.939642+0000","last_peered":"2024-09-15T21:11:46.939642+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:46.939642+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21173516,"num_objects":2585,"num_object_clones":0,"num_object_copies":10340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":219}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10272","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.920002+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:46.920002+0000","last_peered":"2024-09-15T21:11:46.920002+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:46.920002+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20387084,"num_objects":2489,"num_object_clones":0,"num_object_copies":9956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10270,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":15}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'9990","reported_seq":10056,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:48.007095+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:48.007095+0000","last_peered":"2024-09-15T21:11:48.007095+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:48.007095+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20163096,"num_objects":2462,"num_object_clones":0,"num_object_copies":9848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":130}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10103","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.859140+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:49.859140+0000","last_peered":"2024-09-15T21:11:49.859140+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:49.859140+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141324,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9897","reported_seq":14566,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:48.018902+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:48.018902+0000","last_peered":"2024-09-15T21:11:48.018902+0000","last_clean":"2024-09-15T21:11:48.018902+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:48.018902+0000","last_undegraded":"2024-09-15T21:11:48.018902+0000","last_fullsized":"2024-09-15T21:11:48.018902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9897,"log_dups_size":0,"ondisk_log_size":9897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20176896,"num_objects":2463,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'9950","reported_seq":9995,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:48.013211+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:48.013211+0000","last_peered":"2024-09-15T21:11:48.013211+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:48.013211+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9950,"log_dups_size":0,"ondisk_log_size":9950,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20290182,"num_objects":2477,"num_object_clones":0,"num_object_copies":9908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9949,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":221}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9994","reported_seq":11507,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.875959+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:49.875959+0000","last_peered":"2024-09-15T21:11:49.875959+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:49.875959+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535942,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'9968","reported_seq":10030,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:48.091354+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:48.091354+0000","last_peered":"2024-09-15T21:11:48.091354+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:48.091354+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9968,"log_dups_size":0,"ondisk_log_size":9968,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20182284,"num_objects":2464,"num_object_clones":0,"num_object_copies":9856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9966,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":185}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10099","reported_seq":10188,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.831081+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:49.831081+0000","last_peered":"2024-09-15T21:11:49.831081+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:49.831081+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10059","reported_seq":10133,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:46.792335+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:46.792335+0000","last_peered":"2024-09-15T21:11:46.792335+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:46.792335+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20395276,"num_objects":2490,"num_object_clones":0,"num_object_copies":9960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10057,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":326345250,"num_objects":39842,"num_object_clones":0,"num_object_copies":159368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39842,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145236,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":654082048,"data_stored":654082048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159981,"ondisk_log_size":159981,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741604,"kb_used_data":160136,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93630236,"statfs":{"total":96636764160,"available":95877361664,"internally_reserved":0,"allocated":163979264,"data_stored":163806162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":595390160},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,2,5,7,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731504,"kb_used_data":158880,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93640336,"statfs":{"total":96636764160,"available":95887704064,"internally_reserved":0,"allocated":162693120,"data_stored":162520018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,3,1,0,7,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051678,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":598792,"kb_used_data":161784,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93773048,"statfs":{"total":96636764160,"available":96023601152,"internally_reserved":0,"allocated":165666816,"data_stored":165493714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,0,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247916,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752188,"kb_used_data":161068,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93619652,"statfs":{"total":96636764160,"available":95866523648,"internally_reserved":0,"allocated":164933632,"data_stored":164760530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,1,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":164245504,"data_stored":164245504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164765696,"data_stored":164765696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161853440,"data_stored":161853440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":163217408,"data_stored":163217408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:54.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:54.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:54.669 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:54.689 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-15T21:11:54.128921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320283170,"num_objects":39102,"num_object_clones":0,"num_object_copies":156408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145976,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160221,"ondisk_log_size":160221,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2791240,"kb_used_data":629832,"kb_used_omap":14,"kb_used_meta":2161329,"kb_avail":374696120,"statfs":{"total":386547056640,"available":383688826880,"internally_reserved":0,"allocated":644947968,"data_stored":644255560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2213201458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,4,4,7,13,19,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011358"},"pg_stats":[{"pgid":"1.f","version":"35'9938","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.145761+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:53.145761+0000","last_peered":"2024-09-15T21:11:53.145761+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:53.145761+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10125","reported_seq":15018,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:51.398068+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:51.398068+0000","last_peered":"2024-09-15T21:11:51.398068+0000","last_clean":"2024-09-15T21:11:51.398068+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:51.398068+0000","last_undegraded":"2024-09-15T21:11:51.398068+0000","last_fullsized":"2024-09-15T21:11:51.398068+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19865600,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9940","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.667300+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:49.667300+0000","last_peered":"2024-09-15T21:11:49.667300+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:49.667300+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20413086,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9939,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10091","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.827701+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:51.827701+0000","last_peered":"2024-09-15T21:11:51.827701+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:51.827701+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10201","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.141945+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:53.141945+0000","last_peered":"2024-09-15T21:11:53.141945+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:53.141945+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133132,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9990","reported_seq":10040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.105271+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:53.105271+0000","last_peered":"2024-09-15T21:11:53.105271+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:53.105271+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19546112,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9990,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10239","reported_seq":10315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.828569+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:51.828569+0000","last_peered":"2024-09-15T21:11:51.828569+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:51.828569+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20747532,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10329","reported_seq":10404,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.926108+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:51.926108+0000","last_peered":"2024-09-15T21:11:51.926108+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:51.926108+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19920140,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10057","reported_seq":10123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.023217+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:53.023217+0000","last_peered":"2024-09-15T21:11:53.023217+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:53.023217+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19614232,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":63}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10103","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.859140+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:49.859140+0000","last_peered":"2024-09-15T21:11:49.859140+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:49.859140+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141324,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9962","reported_seq":14631,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:53.149863+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:53.149863+0000","last_peered":"2024-09-15T21:11:53.149863+0000","last_clean":"2024-09-15T21:11:53.149863+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:53.149863+0000","last_undegraded":"2024-09-15T21:11:53.149863+0000","last_fullsized":"2024-09-15T21:11:53.149863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10020","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.180417+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:53.180417+0000","last_peered":"2024-09-15T21:11:53.180417+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:53.180417+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19716742,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10019,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9994","reported_seq":11507,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.875959+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:49.875959+0000","last_peered":"2024-09-15T21:11:49.875959+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:49.875959+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535942,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10027","reported_seq":10089,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.146609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:53.146609+0000","last_peered":"2024-09-15T21:11:53.146609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:53.146609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19698956,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10099","reported_seq":10188,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.831081+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:49.831081+0000","last_peered":"2024-09-15T21:11:49.831081+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:49.831081+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10106","reported_seq":10180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.792891+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:51.792891+0000","last_peered":"2024-09-15T21:11:51.792891+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:51.792891+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010252,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":320283170,"num_objects":39102,"num_object_clones":0,"num_object_copies":156408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145976,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":641806336,"data_stored":641806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160221,"ondisk_log_size":160221,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725108,"kb_used_data":156132,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93646732,"statfs":{"total":96636764160,"available":95894253568,"internally_reserved":0,"allocated":159879168,"data_stored":159706066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731504,"kb_used_data":158880,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93640336,"statfs":{"total":96636764160,"available":95887704064,"internally_reserved":0,"allocated":162693120,"data_stored":162520018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,3,1,0,7,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582244,"kb_used_data":157716,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93789596,"statfs":{"total":96636764160,"available":96040546304,"internally_reserved":0,"allocated":161501184,"data_stored":161328082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,3,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752384,"kb_used_data":157104,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93619456,"statfs":{"total":96636764160,"available":95866322944,"internally_reserved":0,"allocated":160874496,"data_stored":160701394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160178176,"data_stored":160178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160702464,"data_stored":160702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161853440,"data_stored":161853440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159072256,"data_stored":159072256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:54.690 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:55.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:55.253 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:55.277 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":316,"stamp":"2024-09-15T21:11:54.128921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":320283170,"num_objects":39102,"num_object_clones":0,"num_object_copies":156408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145976,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160221,"ondisk_log_size":160221,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2791240,"kb_used_data":629832,"kb_used_omap":14,"kb_used_meta":2161329,"kb_avail":374696120,"statfs":{"total":386547056640,"available":383688826880,"internally_reserved":0,"allocated":644947968,"data_stored":644255560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2213201458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,4,4,7,13,19,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19906560,"num_objects":-2430,"num_object_clones":0,"num_object_copies":-9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011358"},"pg_stats":[{"pgid":"1.f","version":"35'9938","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.145761+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:53.145761+0000","last_peered":"2024-09-15T21:11:53.145761+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:53.145761+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10125","reported_seq":15018,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:51.398068+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:51.398068+0000","last_peered":"2024-09-15T21:11:51.398068+0000","last_clean":"2024-09-15T21:11:51.398068+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:51.398068+0000","last_undegraded":"2024-09-15T21:11:51.398068+0000","last_fullsized":"2024-09-15T21:11:51.398068+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19865600,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9940","reported_seq":10026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.667300+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:49.667300+0000","last_peered":"2024-09-15T21:11:49.667300+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:49.667300+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9940,"log_dups_size":0,"ondisk_log_size":9940,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20413086,"num_objects":2493,"num_object_clones":0,"num_object_copies":9972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9939,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":235}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10091","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.827701+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:51.827701+0000","last_peered":"2024-09-15T21:11:51.827701+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:51.827701+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10201","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.141945+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:53.141945+0000","last_peered":"2024-09-15T21:11:53.141945+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:53.141945+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133132,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9990","reported_seq":10040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.105271+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:53.105271+0000","last_peered":"2024-09-15T21:11:53.105271+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:53.105271+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19546112,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9990,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10239","reported_seq":10315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.828569+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:51.828569+0000","last_peered":"2024-09-15T21:11:51.828569+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:51.828569+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20747532,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10329","reported_seq":10404,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.926108+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:51.926108+0000","last_peered":"2024-09-15T21:11:51.926108+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:51.926108+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19920140,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10057","reported_seq":10123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.023217+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:53.023217+0000","last_peered":"2024-09-15T21:11:53.023217+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:53.023217+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19614232,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":63}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10103","reported_seq":10189,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.859140+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:49.859140+0000","last_peered":"2024-09-15T21:11:49.859140+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:49.859140+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20141324,"num_objects":2459,"num_object_clones":0,"num_object_copies":9836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10101,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":124}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9962","reported_seq":14631,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:53.149863+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:53.149863+0000","last_peered":"2024-09-15T21:11:53.149863+0000","last_clean":"2024-09-15T21:11:53.149863+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:53.149863+0000","last_undegraded":"2024-09-15T21:11:53.149863+0000","last_fullsized":"2024-09-15T21:11:53.149863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10020","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.180417+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:53.180417+0000","last_peered":"2024-09-15T21:11:53.180417+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:53.180417+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19716742,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10019,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'9994","reported_seq":11507,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.875959+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:49.875959+0000","last_peered":"2024-09-15T21:11:49.875959+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:49.875959+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9994,"log_dups_size":0,"ondisk_log_size":9994,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20535942,"num_objects":2507,"num_object_clones":0,"num_object_copies":10028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9993,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":715,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":934}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10027","reported_seq":10089,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.146609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:53.146609+0000","last_peered":"2024-09-15T21:11:53.146609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:53.146609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19698956,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10099","reported_seq":10188,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:49.831081+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:49.831081+0000","last_peered":"2024-09-15T21:11:49.831081+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:49.831081+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21148940,"num_objects":2582,"num_object_clones":0,"num_object_copies":10328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10097,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":301}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10106","reported_seq":10180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.792891+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:51.792891+0000","last_peered":"2024-09-15T21:11:51.792891+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:51.792891+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010252,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":320283170,"num_objects":39102,"num_object_clones":0,"num_object_copies":156408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31727,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145976,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":641806336,"data_stored":641806336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160221,"ondisk_log_size":160221,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725108,"kb_used_data":156132,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93646732,"statfs":{"total":96636764160,"available":95894253568,"internally_reserved":0,"allocated":159879168,"data_stored":159706066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731504,"kb_used_data":158880,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93640336,"statfs":{"total":96636764160,"available":95887704064,"internally_reserved":0,"allocated":162693120,"data_stored":162520018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,3,1,0,7,1,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582244,"kb_used_data":157716,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93789596,"statfs":{"total":96636764160,"available":96040546304,"internally_reserved":0,"allocated":161501184,"data_stored":161328082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,3,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752384,"kb_used_data":157104,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93619456,"statfs":{"total":96636764160,"available":95866322944,"internally_reserved":0,"allocated":160874496,"data_stored":160701394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160178176,"data_stored":160178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160702464,"data_stored":160702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":161853440,"data_stored":161853440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159072256,"data_stored":159072256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'9938", 'reported_seq': 9989, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.145761+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:53.145761+0000', 'last_peered': '2024-09-15T21:11:53.145761+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:53.145761+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9938, 'log_dups_size': 0, 'ondisk_log_size': 9938, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19775488, 'num_objects': 2414, 'num_object_clones': 0, 'num_object_copies': 9656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2414, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9938, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 138}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'9940", 'reported_seq': 10026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:49.667300+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:49.667300+0000', 'last_peered': '2024-09-15T21:11:49.667300+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:49.667300+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9940, 'log_dups_size': 0, 'ondisk_log_size': 9940, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20413086, 'num_objects': 2493, 'num_object_clones': 0, 'num_object_copies': 9972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2493, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 9939, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 235}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10091", 'reported_seq': 10166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:51.827701+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:51.827701+0000', 'last_peered': '2024-09-15T21:11:51.827701+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:51.827701+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19371276, 'num_objects': 2365, 'num_object_clones': 0, 'num_object_copies': 9460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10089, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 31}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:55.278 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10201", 'reported_seq': 10263, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.141945+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:53.141945+0000', 'last_peered': '2024-09-15T21:11:53.141945+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:53.141945+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20133132, 'num_objects': 2458, 'num_object_clones': 0, 'num_object_copies': 9832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10199, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 147}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'9990", 'reported_seq': 10040, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.105271+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:53.105271+0000', 'last_peered': '2024-09-15T21:11:53.105271+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:53.105271+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9990, 'log_dups_size': 0, 'ondisk_log_size': 9990, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19546112, 'num_objects': 2386, 'num_object_clones': 0, 'num_object_copies': 9544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2386, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9990, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 70}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10239", 'reported_seq': 10315, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:51.828569+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:51.828569+0000', 'last_peered': '2024-09-15T21:11:51.828569+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:51.828569+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20747532, 'num_objects': 2533, 'num_object_clones': 0, 'num_object_copies': 10132, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2533, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10237, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 167}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10329", 'reported_seq': 10404, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:51.926108+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:51.926108+0000', 'last_peered': '2024-09-15T21:11:51.926108+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:51.926108+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19920140, 'num_objects': 2432, 'num_object_clones': 0, 'num_object_copies': 9728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2432, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10327, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -42}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10057", 'reported_seq': 10123, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.023217+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:53.023217+0000', 'last_peered': '2024-09-15T21:11:53.023217+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:53.023217+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19614232, 'num_objects': 2395, 'num_object_clones': 0, 'num_object_copies': 9580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2395, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10054, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 63}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:11:55.279 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10103", 'reported_seq': 10189, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:49.859140+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:49.859140+0000', 'last_peered': '2024-09-15T21:11:49.859140+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:49.859140+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20141324, 'num_objects': 2459, 'num_object_clones': 0, 'num_object_copies': 9836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2459, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10101, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 124}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10020", 'reported_seq': 10064, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.180417+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:53.180417+0000', 'last_peered': '2024-09-15T21:11:53.180417+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:53.180417+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19716742, 'num_objects': 2407, 'num_object_clones': 0, 'num_object_copies': 9628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10019, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 151}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'9994", 'reported_seq': 11507, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:49.875959+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:49.875959+0000', 'last_peered': '2024-09-15T21:11:49.875959+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:49.875959+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 9994, 'log_dups_size': 0, 'ondisk_log_size': 9994, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20535942, 'num_objects': 2507, 'num_object_clones': 0, 'num_object_copies': 10028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1573, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2507, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 9993, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 715, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10027", 'reported_seq': 10089, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:53.146609+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:53.146609+0000', 'last_peered': '2024-09-15T21:11:53.146609+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:53.146609+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19698956, 'num_objects': 2405, 'num_object_clones': 0, 'num_object_copies': 9620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2405, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10025, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10099", 'reported_seq': 10188, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:49.831081+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:49.831081+0000', 'last_peered': '2024-09-15T21:11:49.831081+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:49.831081+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21148940, 'num_objects': 2582, 'num_object_clones': 0, 'num_object_copies': 10328, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2582, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10097, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 301}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:11:55.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10106", 'reported_seq': 10180, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:51.792891+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:51.792891+0000', 'last_peered': '2024-09-15T21:11:51.792891+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:51.792891+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20010252, 'num_objects': 2443, 'num_object_clones': 0, 'num_object_copies': 9772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2443, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10104, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:11:55.281 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:11:56.027 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:56.028 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31727/156408 objects degraded (20.285%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":581,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":39102,"data_bytes":320283170,"bytes_used":2858229760,"bytes_avail":383688826880,"bytes_total":386547056640,"degraded_objects":31727,"degraded_total":156408,"degraded_ratio":0.20284768042555368,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":202},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:11:56.050 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:11:56.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:11:56.598 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:11:56.599 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:11:56.630 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-15T21:11:56.129731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":318497314,"num_objects":38884,"num_object_clones":0,"num_object_copies":155536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146194,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160339,"ondisk_log_size":160339,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2791444,"kb_used_data":625860,"kb_used_omap":14,"kb_used_meta":2165489,"kb_avail":374695916,"statfs":{"total":386547056640,"available":383688617984,"internally_reserved":0,"allocated":640880640,"data_stored":640188232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2217461298},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,3,8,12,21,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17997824,"num_objects":-2197,"num_object_clones":0,"num_object_copies":-8788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-147,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008621"},"pg_stats":[{"pgid":"1.f","version":"35'9938","reported_seq":9989,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.145761+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:53.145761+0000","last_peered":"2024-09-15T21:11:53.145761+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:53.145761+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9938,"log_dups_size":0,"ondisk_log_size":9938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19775488,"num_objects":2414,"num_object_clones":0,"num_object_copies":9656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9938,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":138}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10125","reported_seq":15018,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:51.398068+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:51.398068+0000","last_peered":"2024-09-15T21:11:51.398068+0000","last_clean":"2024-09-15T21:11:51.398068+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:51.398068+0000","last_undegraded":"2024-09-15T21:11:51.398068+0000","last_fullsized":"2024-09-15T21:11:51.398068+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19865600,"num_objects":2425,"num_object_clones":0,"num_object_copies":9700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'9989","reported_seq":10075,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:54.722688+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:54.722688+0000","last_peered":"2024-09-15T21:11:54.722688+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:54.722688+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9989,"log_dups_size":0,"ondisk_log_size":9989,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20011678,"num_objects":2444,"num_object_clones":0,"num_object_copies":9776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9988,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":186}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10091","reported_seq":10166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.827701+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:51.827701+0000","last_peered":"2024-09-15T21:11:51.827701+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:51.827701+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10089,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":31}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10201","reported_seq":10263,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.141945+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:53.141945+0000","last_peered":"2024-09-15T21:11:53.141945+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:53.141945+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20133132,"num_objects":2458,"num_object_clones":0,"num_object_copies":9832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":147}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'9990","reported_seq":10040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.105271+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:53.105271+0000","last_peered":"2024-09-15T21:11:53.105271+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:53.105271+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":9990,"log_dups_size":0,"ondisk_log_size":9990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19546112,"num_objects":2386,"num_object_clones":0,"num_object_copies":9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9990,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":70}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10239","reported_seq":10315,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.828569+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:51.828569+0000","last_peered":"2024-09-15T21:11:51.828569+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:51.828569+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20747532,"num_objects":2533,"num_object_clones":0,"num_object_copies":10132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10237,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":167}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10329","reported_seq":10404,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.926108+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:51.926108+0000","last_peered":"2024-09-15T21:11:51.926108+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:51.926108+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19920140,"num_objects":2432,"num_object_clones":0,"num_object_copies":9728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-42}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10057","reported_seq":10123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.023217+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:53.023217+0000","last_peered":"2024-09-15T21:11:53.023217+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:53.023217+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19614232,"num_objects":2395,"num_object_clones":0,"num_object_copies":9580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10054,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":63}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10157","reported_seq":10244,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:54.716419+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:54.716419+0000","last_peered":"2024-09-15T21:11:54.716419+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:54.716419+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19698956,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10155,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":70}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'9962","reported_seq":14631,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:53.149863+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:53.149863+0000","last_peered":"2024-09-15T21:11:53.149863+0000","last_clean":"2024-09-15T21:11:53.149863+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:53.149863+0000","last_undegraded":"2024-09-15T21:11:53.149863+0000","last_fullsized":"2024-09-15T21:11:53.149863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":9962,"log_dups_size":0,"ondisk_log_size":9962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19644416,"num_objects":2398,"num_object_clones":0,"num_object_copies":9592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4073,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10020","reported_seq":10064,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.180417+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:53.180417+0000","last_peered":"2024-09-15T21:11:53.180417+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:53.180417+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19716742,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10019,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":151}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10052","reported_seq":11667,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:54.813259+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:54.813259+0000","last_peered":"2024-09-15T21:11:54.813259+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:54.813259+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20060806,"num_objects":2449,"num_object_clones":0,"num_object_copies":9796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1523,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10051,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":926}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10027","reported_seq":10089,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:53.146609+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:53.146609+0000","last_peered":"2024-09-15T21:11:53.146609+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:53.146609+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19698956,"num_objects":2405,"num_object_clones":0,"num_object_copies":9620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10025,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10156","reported_seq":10245,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:54.646951+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:54.646951+0000","last_peered":"2024-09-15T21:11:54.646951+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:54.646951+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20681996,"num_objects":2525,"num_object_clones":0,"num_object_copies":10100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10154,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":244}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10106","reported_seq":10180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:51.792891+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:51.792891+0000","last_peered":"2024-09-15T21:11:51.792891+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:51.792891+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20010252,"num_objects":2443,"num_object_clones":0,"num_object_copies":9772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10104,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":318497314,"num_objects":38884,"num_object_clones":0,"num_object_copies":155536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31677,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38884,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146194,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":786,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":637685760,"data_stored":637685760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160339,"ondisk_log_size":160339,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725108,"kb_used_data":156132,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93646732,"statfs":{"total":96636764160,"available":95894253568,"internally_reserved":0,"allocated":159879168,"data_stored":159706066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4,2,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731708,"kb_used_data":154908,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93640132,"statfs":{"total":96636764160,"available":95887495168,"internally_reserved":0,"allocated":158625792,"data_stored":158452690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,6,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051679,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582244,"kb_used_data":157716,"kb_used_omap":4,"kb_used_meta":424507,"kb_avail":93789596,"statfs":{"total":96636764160,"available":96040546304,"internally_reserved":0,"allocated":161501184,"data_stored":161328082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":434695887},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,3,7,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247917,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752384,"kb_used_data":157104,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93619456,"statfs":{"total":96636764160,"available":95866322944,"internally_reserved":0,"allocated":160874496,"data_stored":160701394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":160178176,"data_stored":160178176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":160702464,"data_stored":160702464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":157732864,"data_stored":157732864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":159072256,"data_stored":159072256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:11:59.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:00.254 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:00.254 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:00.281 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-15T21:12:00.133117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":310215202,"num_objects":37873,"num_object_clones":0,"num_object_copies":151492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147205,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792460,"kb_used_data":610252,"kb_used_omap":14,"kb_used_meta":2182129,"kb_avail":374694900,"statfs":{"total":386547056640,"available":383687577600,"internally_reserved":0,"allocated":624898048,"data_stored":624205640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2234500658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,16,14,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21954560,"num_objects":-2680,"num_object_clones":0,"num_object_copies":-10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010473"},"pg_stats":[{"pgid":"1.f","version":"35'10012","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.383582+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:58.383582+0000","last_peered":"2024-09-15T21:11:58.383582+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:58.383582+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10190","reported_seq":15082,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:56.530274+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:56.530274+0000","last_peered":"2024-09-15T21:11:56.530274+0000","last_clean":"2024-09-15T21:11:56.530274+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:56.530274+0000","last_undegraded":"2024-09-15T21:11:56.530274+0000","last_fullsized":"2024-09-15T21:11:56.530274+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10064","reported_seq":10149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.779350+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:59.779350+0000","last_peered":"2024-09-15T21:11:59.779350+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:59.779350+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19397278,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10063,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10150","reported_seq":10225,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.903718+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:56.903718+0000","last_peered":"2024-09-15T21:11:56.903718+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:56.903718+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18887948,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10271","reported_seq":10333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.319458+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:58.319458+0000","last_peered":"2024-09-15T21:11:58.319458+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:58.319458+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19559692,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10056","reported_seq":10106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.246710+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:58.246710+0000","last_peered":"2024-09-15T21:11:58.246710+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:58.246710+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10056,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10302","reported_seq":10378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.827580+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:56.827580+0000","last_peered":"2024-09-15T21:11:56.827580+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:56.827580+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231436,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10396","reported_seq":10471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.972393+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:56.972393+0000","last_peered":"2024-09-15T21:11:56.972393+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:56.972393+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10394,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10117","reported_seq":10182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.312079+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:58.312079+0000","last_peered":"2024-09-15T21:11:58.312079+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:58.312079+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19122712,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10210","reported_seq":10296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.879022+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:59.879022+0000","last_peered":"2024-09-15T21:11:59.879022+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:59.879022+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264780,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10021","reported_seq":14690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:58.281227+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:58.281227+0000","last_peered":"2024-09-15T21:11:58.281227+0000","last_clean":"2024-09-15T21:11:58.281227+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:58.281227+0000","last_undegraded":"2024-09-15T21:11:58.281227+0000","last_fullsized":"2024-09-15T21:11:58.281227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10072","reported_seq":10117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.395158+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:58.395158+0000","last_peered":"2024-09-15T21:11:58.395158+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:58.395158+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19290758,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10121","reported_seq":11837,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.985935+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:59.985935+0000","last_peered":"2024-09-15T21:11:59.985935+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:59.985935+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495558,"num_objects":2380,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10090","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.157487+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:58.157487+0000","last_peered":"2024-09-15T21:11:58.157487+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:58.157487+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182860,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10202","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.964928+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:59.964928+0000","last_peered":"2024-09-15T21:11:59.964928+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:59.964928+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305164,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10176","reported_seq":10251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:57.000162+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:57.000162+0000","last_peered":"2024-09-15T21:11:57.000162+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:57.000162+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19436812,"num_objects":2373,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":310215202,"num_objects":37873,"num_object_clones":0,"num_object_copies":151492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147205,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":621449216,"data_stored":621449216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725368,"kb_used_data":152232,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93646472,"statfs":{"total":96636764160,"available":95893987328,"internally_reserved":0,"allocated":155885568,"data_stored":155712466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731976,"kb_used_data":151032,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93639864,"statfs":{"total":96636764160,"available":95887220736,"internally_reserved":0,"allocated":154656768,"data_stored":154483666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582488,"kb_used_data":153800,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93789352,"statfs":{"total":96636764160,"available":96040296448,"internally_reserved":0,"allocated":157491200,"data_stored":157318098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752628,"kb_used_data":153188,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93619212,"statfs":{"total":96636764160,"available":95866073088,"internally_reserved":0,"allocated":156864512,"data_stored":156691410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,4,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156164096,"data_stored":156164096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156631040,"data_stored":156631040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":154972160,"data_stored":154972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:00.282 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:00.968 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:00.968 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:00.992 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-15T21:12:00.133117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":310215202,"num_objects":37873,"num_object_clones":0,"num_object_copies":151492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147205,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792460,"kb_used_data":610252,"kb_used_omap":14,"kb_used_meta":2182129,"kb_avail":374694900,"statfs":{"total":386547056640,"available":383687577600,"internally_reserved":0,"allocated":624898048,"data_stored":624205640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2234500658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,16,14,5,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21954560,"num_objects":-2680,"num_object_clones":0,"num_object_copies":-10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2680,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010473"},"pg_stats":[{"pgid":"1.f","version":"35'10012","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.383582+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:58.383582+0000","last_peered":"2024-09-15T21:11:58.383582+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:58.383582+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10190","reported_seq":15082,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:56.530274+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:11:56.530274+0000","last_peered":"2024-09-15T21:11:56.530274+0000","last_clean":"2024-09-15T21:11:56.530274+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:11:56.530274+0000","last_undegraded":"2024-09-15T21:11:56.530274+0000","last_fullsized":"2024-09-15T21:11:56.530274+0000","mapping_epoch":34,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":2360,"num_object_clones":0,"num_object_copies":9440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10064","reported_seq":10149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.779350+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:59.779350+0000","last_peered":"2024-09-15T21:11:59.779350+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:59.779350+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19397278,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10063,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10150","reported_seq":10225,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.903718+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:56.903718+0000","last_peered":"2024-09-15T21:11:56.903718+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:56.903718+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18887948,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10271","reported_seq":10333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.319458+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:58.319458+0000","last_peered":"2024-09-15T21:11:58.319458+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:58.319458+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19559692,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10056","reported_seq":10106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.246710+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:58.246710+0000","last_peered":"2024-09-15T21:11:58.246710+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:58.246710+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10056,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10302","reported_seq":10378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.827580+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:56.827580+0000","last_peered":"2024-09-15T21:11:56.827580+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:56.827580+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231436,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10396","reported_seq":10471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.972393+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:56.972393+0000","last_peered":"2024-09-15T21:11:56.972393+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:56.972393+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10394,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10117","reported_seq":10182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.312079+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:58.312079+0000","last_peered":"2024-09-15T21:11:58.312079+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:58.312079+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19122712,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10210","reported_seq":10296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.879022+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:59.879022+0000","last_peered":"2024-09-15T21:11:59.879022+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:59.879022+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264780,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10021","reported_seq":14690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:58.281227+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:58.281227+0000","last_peered":"2024-09-15T21:11:58.281227+0000","last_clean":"2024-09-15T21:11:58.281227+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:58.281227+0000","last_undegraded":"2024-09-15T21:11:58.281227+0000","last_fullsized":"2024-09-15T21:11:58.281227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10072","reported_seq":10117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.395158+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:58.395158+0000","last_peered":"2024-09-15T21:11:58.395158+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:58.395158+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19290758,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10121","reported_seq":11837,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.985935+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:59.985935+0000","last_peered":"2024-09-15T21:11:59.985935+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:59.985935+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495558,"num_objects":2380,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10090","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.157487+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:58.157487+0000","last_peered":"2024-09-15T21:11:58.157487+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:58.157487+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182860,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10202","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.964928+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:59.964928+0000","last_peered":"2024-09-15T21:11:59.964928+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:59.964928+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305164,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10176","reported_seq":10251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:57.000162+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:57.000162+0000","last_peered":"2024-09-15T21:11:57.000162+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:57.000162+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19436812,"num_objects":2373,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":310215202,"num_objects":37873,"num_object_clones":0,"num_object_copies":151492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147205,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":621449216,"data_stored":621449216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160750,"ondisk_log_size":160750,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725368,"kb_used_data":152232,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93646472,"statfs":{"total":96636764160,"available":95893987328,"internally_reserved":0,"allocated":155885568,"data_stored":155712466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731976,"kb_used_data":151032,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93639864,"statfs":{"total":96636764160,"available":95887220736,"internally_reserved":0,"allocated":154656768,"data_stored":154483666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051680,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582488,"kb_used_data":153800,"kb_used_omap":4,"kb_used_meta":428667,"kb_avail":93789352,"statfs":{"total":96636764160,"available":96040296448,"internally_reserved":0,"allocated":157491200,"data_stored":157318098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":438955727},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752628,"kb_used_data":153188,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93619212,"statfs":{"total":96636764160,"available":95866073088,"internally_reserved":0,"allocated":156864512,"data_stored":156691410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,4,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156164096,"data_stored":156164096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":156631040,"data_stored":156631040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":154972160,"data_stored":154972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10012", 'reported_seq': 10063, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.383582+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:11:58.383582+0000', 'last_peered': '2024-09-15T21:11:58.383582+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:11:58.383582+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19169280, 'num_objects': 2340, 'num_object_clones': 0, 'num_object_copies': 9360, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2340, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10012, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 64}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10064", 'reported_seq': 10149, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:59.779350+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:11:59.779350+0000', 'last_peered': '2024-09-15T21:11:59.779350+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:11:59.779350+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19397278, 'num_objects': 2369, 'num_object_clones': 0, 'num_object_copies': 9476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2369, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10063, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 111}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10150", 'reported_seq': 10225, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:56.903718+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:11:56.903718+0000', 'last_peered': '2024-09-15T21:11:56.903718+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:11:56.903718+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18887948, 'num_objects': 2306, 'num_object_clones': 0, 'num_object_copies': 9224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2306, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10148, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -28}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:00.993 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10271", 'reported_seq': 10333, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.319458+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:11:58.319458+0000', 'last_peered': '2024-09-15T21:11:58.319458+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:11:58.319458+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19559692, 'num_objects': 2388, 'num_object_clones': 0, 'num_object_copies': 9552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2388, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10269, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 77}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10056", 'reported_seq': 10106, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.246710+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:11:58.246710+0000', 'last_peered': '2024-09-15T21:11:58.246710+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:11:58.246710+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19005440, 'num_objects': 2320, 'num_object_clones': 0, 'num_object_copies': 9280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2320, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10056, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10302", 'reported_seq': 10378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:56.827580+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:11:56.827580+0000', 'last_peered': '2024-09-15T21:11:56.827580+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:11:56.827580+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20231436, 'num_objects': 2470, 'num_object_clones': 0, 'num_object_copies': 9880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2470, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10300, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 104}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10396", 'reported_seq': 10471, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:56.972393+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:11:56.972393+0000', 'last_peered': '2024-09-15T21:11:56.972393+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:11:56.972393+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19371276, 'num_objects': 2365, 'num_object_clones': 0, 'num_object_copies': 9460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2365, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10394, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -109}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10117", 'reported_seq': 10182, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.312079+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:11:58.312079+0000', 'last_peered': '2024-09-15T21:11:58.312079+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:11:58.312079+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19122712, 'num_objects': 2335, 'num_object_clones': 0, 'num_object_copies': 9340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10114, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 3}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10210", 'reported_seq': 10296, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:59.879022+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:11:59.879022+0000', 'last_peered': '2024-09-15T21:11:59.879022+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:11:59.879022+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19264780, 'num_objects': 2352, 'num_object_clones': 0, 'num_object_copies': 9408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10208, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': 17}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:00.994 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10072", 'reported_seq': 10117, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.395158+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:11:58.395158+0000', 'last_peered': '2024-09-15T21:11:58.395158+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:11:58.395158+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19290758, 'num_objects': 2355, 'num_object_clones': 0, 'num_object_copies': 9420, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2355, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10071, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 99}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10121", 'reported_seq': 11837, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:59.985935+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:11:59.985935+0000', 'last_peered': '2024-09-15T21:11:59.985935+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:11:59.985935+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19495558, 'num_objects': 2380, 'num_object_clones': 0, 'num_object_copies': 9520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1472, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2380, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10120, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 816, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 908}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10090", 'reported_seq': 10152, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:58.157487+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:11:58.157487+0000', 'last_peered': '2024-09-15T21:11:58.157487+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:11:58.157487+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19182860, 'num_objects': 2342, 'num_object_clones': 0, 'num_object_copies': 9368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2342, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10088, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 63}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10202", 'reported_seq': 10291, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:59.964928+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:11:59.964928+0000', 'last_peered': '2024-09-15T21:11:59.964928+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:11:59.964928+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 20305164, 'num_objects': 2479, 'num_object_clones': 0, 'num_object_copies': 9916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2479, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10200, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:00.995 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10176", 'reported_seq': 10251, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:11:57.000162+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:11:57.000162+0000', 'last_peered': '2024-09-15T21:11:57.000162+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:11:57.000162+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19436812, 'num_objects': 2373, 'num_object_clones': 0, 'num_object_copies': 9492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2373, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10174, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': 58}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:00.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:01.758 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:01.759 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31626/151492 objects degraded (20.876%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":587,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":37873,"data_bytes":310215202,"bytes_used":2859479040,"bytes_avail":383687577600,"bytes_total":386547056640,"degraded_objects":31626,"degraded_total":151492,"degraded_ratio":0.20876349906265682,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":223},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:01.787 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:01.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:02.421 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:02.421 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:02.445 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-15T21:12:02.133832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":309756450,"num_objects":37817,"num_object_clones":0,"num_object_copies":151268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147261,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2792116,"kb_used_data":605724,"kb_used_omap":14,"kb_used_meta":2186289,"kb_avail":374695244,"statfs":{"total":386547056640,"available":383687929856,"internally_reserved":0,"allocated":620261376,"data_stored":619568968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2238760498},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,11,17,10,5,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16588800,"num_objects":-2025,"num_object_clones":0,"num_object_copies":-8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010518"},"pg_stats":[{"pgid":"1.f","version":"35'10012","reported_seq":10063,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.383582+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:11:58.383582+0000","last_peered":"2024-09-15T21:11:58.383582+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:11:58.383582+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19169280,"num_objects":2340,"num_object_clones":0,"num_object_copies":9360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10012,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":64}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10246","reported_seq":15139,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:01.664843+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:01.664843+0000","last_peered":"2024-09-15T21:12:01.664843+0000","last_clean":"2024-09-15T21:12:01.664843+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:01.664843+0000","last_undegraded":"2024-09-15T21:12:01.664843+0000","last_fullsized":"2024-09-15T21:12:01.664843+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10064","reported_seq":10149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.779350+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:59.779350+0000","last_peered":"2024-09-15T21:11:59.779350+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:59.779350+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19397278,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10063,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10150","reported_seq":10225,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.903718+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:11:56.903718+0000","last_peered":"2024-09-15T21:11:56.903718+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:11:56.903718+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18887948,"num_objects":2306,"num_object_clones":0,"num_object_copies":9224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10148,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-28}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10271","reported_seq":10333,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.319458+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:11:58.319458+0000","last_peered":"2024-09-15T21:11:58.319458+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:11:58.319458+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19559692,"num_objects":2388,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10269,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":77}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10056","reported_seq":10106,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.246710+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:11:58.246710+0000","last_peered":"2024-09-15T21:11:58.246710+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:11:58.246710+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":2320,"num_object_clones":0,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10056,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10302","reported_seq":10378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.827580+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:11:56.827580+0000","last_peered":"2024-09-15T21:11:56.827580+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:11:56.827580+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20231436,"num_objects":2470,"num_object_clones":0,"num_object_copies":9880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":104}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10396","reported_seq":10471,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:56.972393+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:11:56.972393+0000","last_peered":"2024-09-15T21:11:56.972393+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:11:56.972393+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19371276,"num_objects":2365,"num_object_clones":0,"num_object_copies":9460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10394,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-109}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10117","reported_seq":10182,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.312079+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:11:58.312079+0000","last_peered":"2024-09-15T21:11:58.312079+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:11:58.312079+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19122712,"num_objects":2335,"num_object_clones":0,"num_object_copies":9340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10114,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10210","reported_seq":10296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.879022+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:59.879022+0000","last_peered":"2024-09-15T21:11:59.879022+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:59.879022+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264780,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10021","reported_seq":14690,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:11:58.281227+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:11:58.281227+0000","last_peered":"2024-09-15T21:11:58.281227+0000","last_clean":"2024-09-15T21:11:58.281227+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:11:58.281227+0000","last_undegraded":"2024-09-15T21:11:58.281227+0000","last_fullsized":"2024-09-15T21:11:58.281227+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19161088,"num_objects":2339,"num_object_clones":0,"num_object_copies":9356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4132,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10072","reported_seq":10117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.395158+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:11:58.395158+0000","last_peered":"2024-09-15T21:11:58.395158+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:11:58.395158+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19290758,"num_objects":2355,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10071,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":99}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10121","reported_seq":11837,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.985935+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:59.985935+0000","last_peered":"2024-09-15T21:11:59.985935+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:59.985935+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495558,"num_objects":2380,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10090","reported_seq":10152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:58.157487+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:11:58.157487+0000","last_peered":"2024-09-15T21:11:58.157487+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:11:58.157487+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19182860,"num_objects":2342,"num_object_clones":0,"num_object_copies":9368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10088,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":63}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10202","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.964928+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:59.964928+0000","last_peered":"2024-09-15T21:11:59.964928+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:59.964928+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305164,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10176","reported_seq":10251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:57.000162+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:11:57.000162+0000","last_peered":"2024-09-15T21:11:57.000162+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:11:57.000162+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19436812,"num_objects":2373,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10174,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":58}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":309756450,"num_objects":37817,"num_object_clones":0,"num_object_copies":151268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37817,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147261,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":617365504,"data_stored":617365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160706,"ondisk_log_size":160706,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725368,"kb_used_data":152232,"kb_used_omap":4,"kb_used_meta":573115,"kb_avail":93646472,"statfs":{"total":96636764160,"available":95893987328,"internally_reserved":0,"allocated":155885568,"data_stored":155712466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":586870480},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731976,"kb_used_data":151032,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93639864,"statfs":{"total":96636764160,"available":95887220736,"internally_reserved":0,"allocated":154656768,"data_stored":154483666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582144,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93789696,"statfs":{"total":96636764160,"available":96040648704,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":752628,"kb_used_data":153188,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93619212,"statfs":{"total":96636764160,"available":95866073088,"internally_reserved":0,"allocated":156864512,"data_stored":156691410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,4,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":156164096,"data_stored":156164096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152547328,"data_stored":152547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":154972160,"data_stored":154972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:05.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:06.045 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:06.046 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:06.075 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-15T21:12:04.134436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":304251426,"num_objects":37145,"num_object_clones":0,"num_object_copies":148580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147933,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160678,"ondisk_log_size":160678,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2776124,"kb_used_data":598052,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374711236,"statfs":{"total":386547056640,"available":383704305664,"internally_reserved":0,"allocated":612405248,"data_stored":611712840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,4,13,13,9,8,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19750912,"num_objects":-2411,"num_object_clones":0,"num_object_copies":-9644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2411,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006331"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.263005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:03.263005+0000","last_peered":"2024-09-15T21:12:03.263005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:03.263005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10246","reported_seq":15139,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:01.664843+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:01.664843+0000","last_peered":"2024-09-15T21:12:01.664843+0000","last_clean":"2024-09-15T21:12:01.664843+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:01.664843+0000","last_undegraded":"2024-09-15T21:12:01.664843+0000","last_fullsized":"2024-09-15T21:12:01.664843+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10064","reported_seq":10149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.779350+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:11:59.779350+0000","last_peered":"2024-09-15T21:11:59.779350+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:11:59.779350+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19397278,"num_objects":2369,"num_object_clones":0,"num_object_copies":9476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2369,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10063,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":111}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10207","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:01.947308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:01.947308+0000","last_peered":"2024-09-15T21:12:01.947308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:01.947308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18421004,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-85}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10330","reported_seq":10386,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.448963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:03.448963+0000","last_peered":"2024-09-15T21:12:03.448963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:03.448963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19076364,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10117","reported_seq":10167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.277528+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:03.277528+0000","last_peered":"2024-09-15T21:12:03.277528+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:03.277528+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18505728,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10362","reported_seq":10438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.002710+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:02.002710+0000","last_peered":"2024-09-15T21:12:02.002710+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:02.002710+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19739916,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10360,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10463","reported_seq":10538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.031347+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:02.031347+0000","last_peered":"2024-09-15T21:12:02.031347+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:02.031347+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822412,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10165","reported_seq":10231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.353854+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:03.353854+0000","last_peered":"2024-09-15T21:12:03.353854+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:03.353854+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18729496,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10210","reported_seq":10296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.879022+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:11:59.879022+0000","last_peered":"2024-09-15T21:11:59.879022+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:11:59.879022+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19264780,"num_objects":2352,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10208,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":17}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10080","reported_seq":14749,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:03.448188+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:03.448188+0000","last_peered":"2024-09-15T21:12:03.448188+0000","last_clean":"2024-09-15T21:12:03.448188+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:03.448188+0000","last_undegraded":"2024-09-15T21:12:03.448188+0000","last_fullsized":"2024-09-15T21:12:03.448188+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4191,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10148","reported_seq":10193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.545282+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:03.545282+0000","last_peered":"2024-09-15T21:12:03.545282+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:03.545282+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18668166,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10121","reported_seq":11837,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.985935+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:11:59.985935+0000","last_peered":"2024-09-15T21:11:59.985935+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:11:59.985935+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19495558,"num_objects":2380,"num_object_clones":0,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10120,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":816,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":908}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10153","reported_seq":10215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.473506+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:03.473506+0000","last_peered":"2024-09-15T21:12:03.473506+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:03.473506+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18666764,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10202","reported_seq":10291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:11:59.964928+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:11:59.964928+0000","last_peered":"2024-09-15T21:11:59.964928+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:11:59.964928+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20305164,"num_objects":2479,"num_object_clones":0,"num_object_copies":9916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10200,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10236","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.073896+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:02.073896+0000","last_peered":"2024-09-15T21:12:02.073896+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:02.073896+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":304251426,"num_objects":37145,"num_object_clones":0,"num_object_copies":148580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37145,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147933,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":837,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":609411072,"data_stored":609411072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160678,"ondisk_log_size":160678,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725772,"kb_used_data":148476,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93646068,"statfs":{"total":96636764160,"available":95893573632,"internally_reserved":0,"allocated":152039424,"data_stored":151866322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731976,"kb_used_data":151032,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93639864,"statfs":{"total":96636764160,"available":95887220736,"internally_reserved":0,"allocated":154656768,"data_stored":154483666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,7,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582144,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93789696,"statfs":{"total":96636764160,"available":96040648704,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736232,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93635608,"statfs":{"total":96636764160,"available":95882862592,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152162304,"data_stored":152162304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152547328,"data_stored":152547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153681920,"data_stored":153681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151019520,"data_stored":151019520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:06.076 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:06.678 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:06.678 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:06.699 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-15T21:12:06.135328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302154274,"num_objects":36889,"num_object_clones":0,"num_object_copies":147556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772376,"kb_used_data":594296,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374714984,"statfs":{"total":386547056640,"available":383708143616,"internally_reserved":0,"allocated":608559104,"data_stored":607866696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,16,6,15,10,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006406"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.263005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:03.263005+0000","last_peered":"2024-09-15T21:12:03.263005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:03.263005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10246","reported_seq":15139,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:01.664843+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:01.664843+0000","last_peered":"2024-09-15T21:12:01.664843+0000","last_clean":"2024-09-15T21:12:01.664843+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:01.664843+0000","last_undegraded":"2024-09-15T21:12:01.664843+0000","last_fullsized":"2024-09-15T21:12:01.664843+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10115","reported_seq":10201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:04.571453+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:04.571453+0000","last_peered":"2024-09-15T21:12:04.571453+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:04.571453+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18979486,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10114,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10207","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:01.947308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:01.947308+0000","last_peered":"2024-09-15T21:12:01.947308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:01.947308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18421004,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-85}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10330","reported_seq":10386,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.448963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:03.448963+0000","last_peered":"2024-09-15T21:12:03.448963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:03.448963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19076364,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10117","reported_seq":10167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.277528+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:03.277528+0000","last_peered":"2024-09-15T21:12:03.277528+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:03.277528+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18505728,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10362","reported_seq":10438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.002710+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:02.002710+0000","last_peered":"2024-09-15T21:12:02.002710+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:02.002710+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19739916,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10360,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10463","reported_seq":10538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.031347+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:02.031347+0000","last_peered":"2024-09-15T21:12:02.031347+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:02.031347+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822412,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10165","reported_seq":10231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.353854+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:03.353854+0000","last_peered":"2024-09-15T21:12:03.353854+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:03.353854+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18729496,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10276","reported_seq":10363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:05.042963+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:05.042963+0000","last_peered":"2024-09-15T21:12:05.042963+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:05.042963+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18724108,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-49}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10080","reported_seq":14749,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:03.448188+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:03.448188+0000","last_peered":"2024-09-15T21:12:03.448188+0000","last_clean":"2024-09-15T21:12:03.448188+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:03.448188+0000","last_undegraded":"2024-09-15T21:12:03.448188+0000","last_fullsized":"2024-09-15T21:12:03.448188+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4191,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10148","reported_seq":10193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.545282+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:03.545282+0000","last_peered":"2024-09-15T21:12:03.545282+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:03.545282+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18668166,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10188","reported_seq":12001,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:05.077341+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:05.077341+0000","last_peered":"2024-09-15T21:12:05.077341+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:05.077341+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18946694,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":864,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10153","reported_seq":10215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.473506+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:03.473506+0000","last_peered":"2024-09-15T21:12:03.473506+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:03.473506+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18666764,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10274","reported_seq":10363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:04.969198+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:04.969198+0000","last_peered":"2024-09-15T21:12:04.969198+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:04.969198+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19715340,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10236","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.073896+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:02.073896+0000","last_peered":"2024-09-15T21:12:02.073896+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:02.073896+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":302154274,"num_objects":36889,"num_object_clones":0,"num_object_copies":147556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605388800,"data_stored":605388800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725772,"kb_used_data":148476,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93646068,"statfs":{"total":96636764160,"available":95893573632,"internally_reserved":0,"allocated":152039424,"data_stored":151866322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728228,"kb_used_data":147276,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93643612,"statfs":{"total":96636764160,"available":95891058688,"internally_reserved":0,"allocated":150810624,"data_stored":150637522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582144,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93789696,"statfs":{"total":96636764160,"available":96040648704,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736232,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93635608,"statfs":{"total":96636764160,"available":95882862592,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152162304,"data_stored":152162304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152547328,"data_stored":152547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149659648,"data_stored":149659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151019520,"data_stored":151019520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:06.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:06.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10074", 'reported_seq': 10125, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.263005+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:03.263005+0000', 'last_peered': '2024-09-15T21:12:03.263005+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:03.263005+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18661376, 'num_objects': 2278, 'num_object_clones': 0, 'num_object_copies': 9112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2278, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10074, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:06.700 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10115", 'reported_seq': 10201, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:04.571453+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:04.571453+0000', 'last_peered': '2024-09-15T21:12:04.571453+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:04.571453+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18979486, 'num_objects': 2318, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2318, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10114, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:06.700 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10207", 'reported_seq': 10282, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:01.947308+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:01.947308+0000', 'last_peered': '2024-09-15T21:12:01.947308+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:01.947308+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18421004, 'num_objects': 2249, 'num_object_clones': 0, 'num_object_copies': 8996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10205, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -85}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10330", 'reported_seq': 10386, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.448963+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:03.448963+0000', 'last_peered': '2024-09-15T21:12:03.448963+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:03.448963+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19076364, 'num_objects': 2329, 'num_object_clones': 0, 'num_object_copies': 9316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2329, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10328, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': 18}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10117", 'reported_seq': 10167, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.277528+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:03.277528+0000', 'last_peered': '2024-09-15T21:12:03.277528+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:03.277528+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18505728, 'num_objects': 2259, 'num_object_clones': 0, 'num_object_copies': 9036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2259, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10117, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -57}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10362", 'reported_seq': 10438, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:02.002710+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:02.002710+0000', 'last_peered': '2024-09-15T21:12:02.002710+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:02.002710+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19739916, 'num_objects': 2410, 'num_object_clones': 0, 'num_object_copies': 9640, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2410, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10360, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': 44}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10463", 'reported_seq': 10538, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:02.031347+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:02.031347+0000', 'last_peered': '2024-09-15T21:12:02.031347+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:02.031347+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18822412, 'num_objects': 2298, 'num_object_clones': 0, 'num_object_copies': 9192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10461, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:06.701 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10165", 'reported_seq': 10231, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.353854+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:03.353854+0000', 'last_peered': '2024-09-15T21:12:03.353854+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:03.353854+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18729496, 'num_objects': 2287, 'num_object_clones': 0, 'num_object_copies': 9148, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2287, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10162, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -45}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10276", 'reported_seq': 10363, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:05.042963+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:05.042963+0000', 'last_peered': '2024-09-15T21:12:05.042963+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:05.042963+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18724108, 'num_objects': 2286, 'num_object_clones': 0, 'num_object_copies': 9144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2286, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10274, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -49}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10148", 'reported_seq': 10193, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.545282+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:03.545282+0000', 'last_peered': '2024-09-15T21:12:03.545282+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:03.545282+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18668166, 'num_objects': 2279, 'num_object_clones': 0, 'num_object_copies': 9116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10147, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 23}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10188", 'reported_seq': 12001, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:05.077341+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:05.077341+0000', 'last_peered': '2024-09-15T21:12:05.077341+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:05.077341+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18946694, 'num_objects': 2313, 'num_object_clones': 0, 'num_object_copies': 9252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1424, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10187, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 864, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 889}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10153", 'reported_seq': 10215, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:03.473506+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:03.473506+0000', 'last_peered': '2024-09-15T21:12:03.473506+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:03.473506+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18666764, 'num_objects': 2279, 'num_object_clones': 0, 'num_object_copies': 9116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2279, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10151, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': 2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:06.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10274", 'reported_seq': 10363, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:04.969198+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:04.969198+0000', 'last_peered': '2024-09-15T21:12:04.969198+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:04.969198+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19715340, 'num_objects': 2407, 'num_object_clones': 0, 'num_object_copies': 9628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10272, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:06.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:06.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10236", 'reported_seq': 10310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:02.073896+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:02.073896+0000', 'last_peered': '2024-09-15T21:12:02.073896+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:02.073896+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18945292, 'num_objects': 2313, 'num_object_clones': 0, 'num_object_copies': 9252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2313, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10234, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:06.703 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:07.432 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:07.432 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31578/147556 objects degraded (21.401%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":593,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":36889,"data_bytes":302154274,"bytes_used":2838913024,"bytes_avail":383708143616,"bytes_total":386547056640,"degraded_objects":31578,"degraded_total":147556,"degraded_ratio":0.21400688552142916,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:07.460 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:07.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:08.013 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:08.013 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:08.031 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-15T21:12:06.135328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":302154274,"num_objects":36889,"num_object_clones":0,"num_object_copies":147556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2772376,"kb_used_data":594296,"kb_used_omap":14,"kb_used_meta":2177969,"kb_avail":374714984,"statfs":{"total":386547056640,"available":383708143616,"internally_reserved":0,"allocated":608559104,"data_stored":607866696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2230240818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,16,6,15,10,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-149,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006406"},"pg_stats":[{"pgid":"1.f","version":"35'10074","reported_seq":10125,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.263005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:03.263005+0000","last_peered":"2024-09-15T21:12:03.263005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:03.263005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18661376,"num_objects":2278,"num_object_clones":0,"num_object_copies":9112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10074,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10246","reported_seq":15139,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:01.664843+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:01.664843+0000","last_peered":"2024-09-15T21:12:01.664843+0000","last_clean":"2024-09-15T21:12:01.664843+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:01.664843+0000","last_undegraded":"2024-09-15T21:12:01.664843+0000","last_fullsized":"2024-09-15T21:12:01.664843+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":2304,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10115","reported_seq":10201,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:04.571453+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:04.571453+0000","last_peered":"2024-09-15T21:12:04.571453+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:04.571453+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18979486,"num_objects":2318,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2318,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10114,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10207","reported_seq":10282,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:01.947308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:01.947308+0000","last_peered":"2024-09-15T21:12:01.947308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:01.947308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18421004,"num_objects":2249,"num_object_clones":0,"num_object_copies":8996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10205,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-85}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10330","reported_seq":10386,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.448963+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:03.448963+0000","last_peered":"2024-09-15T21:12:03.448963+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:03.448963+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19076364,"num_objects":2329,"num_object_clones":0,"num_object_copies":9316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10328,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":18}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10117","reported_seq":10167,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.277528+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:03.277528+0000","last_peered":"2024-09-15T21:12:03.277528+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:03.277528+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18505728,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10117,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-57}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10362","reported_seq":10438,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.002710+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:02.002710+0000","last_peered":"2024-09-15T21:12:02.002710+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:02.002710+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19739916,"num_objects":2410,"num_object_clones":0,"num_object_copies":9640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10360,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10463","reported_seq":10538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.031347+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:02.031347+0000","last_peered":"2024-09-15T21:12:02.031347+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:02.031347+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18822412,"num_objects":2298,"num_object_clones":0,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10165","reported_seq":10231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.353854+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:03.353854+0000","last_peered":"2024-09-15T21:12:03.353854+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:03.353854+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18729496,"num_objects":2287,"num_object_clones":0,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10162,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10276","reported_seq":10363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:05.042963+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:05.042963+0000","last_peered":"2024-09-15T21:12:05.042963+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:05.042963+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18724108,"num_objects":2286,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-49}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10080","reported_seq":14749,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:03.448188+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:03.448188+0000","last_peered":"2024-09-15T21:12:03.448188+0000","last_clean":"2024-09-15T21:12:03.448188+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:03.448188+0000","last_undegraded":"2024-09-15T21:12:03.448188+0000","last_fullsized":"2024-09-15T21:12:03.448188+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18677760,"num_objects":2280,"num_object_clones":0,"num_object_copies":9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4191,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10148","reported_seq":10193,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.545282+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:03.545282+0000","last_peered":"2024-09-15T21:12:03.545282+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:03.545282+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18668166,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10147,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10188","reported_seq":12001,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:05.077341+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:05.077341+0000","last_peered":"2024-09-15T21:12:05.077341+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:05.077341+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18946694,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10187,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":864,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":889}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10153","reported_seq":10215,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:03.473506+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:03.473506+0000","last_peered":"2024-09-15T21:12:03.473506+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:03.473506+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18666764,"num_objects":2279,"num_object_clones":0,"num_object_copies":9116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":2279}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10274","reported_seq":10363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:04.969198+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:04.969198+0000","last_peered":"2024-09-15T21:12:04.969198+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:04.969198+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19715340,"num_objects":2407,"num_object_clones":0,"num_object_copies":9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":126}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10236","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:02.073896+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:02.073896+0000","last_peered":"2024-09-15T21:12:02.073896+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:02.073896+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18945292,"num_objects":2313,"num_object_clones":0,"num_object_copies":9252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10234,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":302154274,"num_objects":36889,"num_object_clones":0,"num_object_copies":147556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36889,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148189,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":885,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":605388800,"data_stored":605388800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160834,"ondisk_log_size":160834,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725772,"kb_used_data":148476,"kb_used_omap":4,"kb_used_meta":577275,"kb_avail":93646068,"statfs":{"total":96636764160,"available":95893573632,"internally_reserved":0,"allocated":152039424,"data_stored":151866322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":591130320},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,3,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728228,"kb_used_data":147276,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93643612,"statfs":{"total":96636764160,"available":95891058688,"internally_reserved":0,"allocated":150810624,"data_stored":150637522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,0,8,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051681,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582144,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":432827,"kb_avail":93789696,"statfs":{"total":96636764160,"available":96040648704,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":443215567},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,1,2,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247919,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736232,"kb_used_data":149272,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93635608,"statfs":{"total":96636764160,"available":95882862592,"internally_reserved":0,"allocated":152854528,"data_stored":152681426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":601026254},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,2,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152162304,"data_stored":152162304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152547328,"data_stored":152547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":149659648,"data_stored":149659648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":151019520,"data_stored":151019520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:11.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:11.642 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:11.642 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:11.667 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-15T21:12:10.137029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":294036002,"num_objects":35898,"num_object_clones":0,"num_object_copies":143592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149180,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2760012,"kb_used_data":577748,"kb_used_omap":14,"kb_used_meta":2182129,"kb_avail":374727348,"statfs":{"total":386547056640,"available":383720804352,"internally_reserved":0,"allocated":591613952,"data_stored":590921544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2234500658},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,9,4,15,10,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21807104,"num_objects":-2662,"num_object_clones":0,"num_object_copies":-10648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2662,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004767"},"pg_stats":[{"pgid":"1.f","version":"35'10135","reported_seq":10186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.392463+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:08.392463+0000","last_peered":"2024-09-15T21:12:08.392463+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:08.392463+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10135,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10291","reported_seq":15184,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:06.502115+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:06.502115+0000","last_peered":"2024-09-15T21:12:06.502115+0000","last_clean":"2024-09-15T21:12:06.502115+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:06.502115+0000","last_undegraded":"2024-09-15T21:12:06.502115+0000","last_fullsized":"2024-09-15T21:12:06.502115+0000","mapping_epoch":34,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18505728,"num_objects":2259,"num_object_clones":0,"num_object_copies":9036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":957,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10183","reported_seq":10269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.099856+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:10.099856+0000","last_peered":"2024-09-15T21:12:10.099856+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:10.099856+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18422430,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10182,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10260","reported_seq":10335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.140103+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:07.140103+0000","last_peered":"2024-09-15T21:12:07.140103+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:07.140103+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986828,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10393","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.495294+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:08.495294+0000","last_peered":"2024-09-15T21:12:08.495294+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:08.495294+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18560268,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10183","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.320046+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:08.320046+0000","last_peered":"2024-09-15T21:12:08.320046+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:08.320046+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17965056,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10434","reported_seq":10510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.103323+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:07.103323+0000","last_peered":"2024-09-15T21:12:07.103323+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:07.103323+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19150092,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10432,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10529","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.073889+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:07.073889+0000","last_peered":"2024-09-15T21:12:07.073889+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:07.073889+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18281740,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10226","reported_seq":10292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.461144+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:08.461144+0000","last_peered":"2024-09-15T21:12:08.461144+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:08.461144+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18229784,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10335","reported_seq":10422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.038126+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:10.038126+0000","last_peered":"2024-09-15T21:12:10.038126+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:10.038126+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240780,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10145","reported_seq":14812,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:08.482113+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:08.482113+0000","last_peered":"2024-09-15T21:12:08.482113+0000","last_clean":"2024-09-15T21:12:08.482113+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:08.482113+0000","last_undegraded":"2024-09-15T21:12:08.482113+0000","last_fullsized":"2024-09-15T21:12:08.482113+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18145280,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10193","reported_seq":10238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.506605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:08.506605+0000","last_peered":"2024-09-15T21:12:08.506605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:08.506605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18299526,"num_objects":2234,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10192,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10254","reported_seq":12179,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.010755+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:10.010755+0000","last_peered":"2024-09-15T21:12:10.010755+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:10.010755+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18406022,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10253,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10218","reported_seq":10281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.306727+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:08.306727+0000","last_peered":"2024-09-15T21:12:08.306727+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:08.306727+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134284,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10347","reported_seq":10433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:09.976769+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:09.976769+0000","last_peered":"2024-09-15T21:12:09.976769+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:09.976769+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117324,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":53}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10299","reported_seq":10374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.273094+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:07.273094+0000","last_peered":"2024-09-15T21:12:07.273094+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:07.273094+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10297,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":294036002,"num_objects":35898,"num_object_clones":0,"num_object_copies":143592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35898,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149180,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589193216,"data_stored":589193216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160925,"ondisk_log_size":160925,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725980,"kb_used_data":144524,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93645860,"statfs":{"total":96636764160,"available":95893360640,"internally_reserved":0,"allocated":147992576,"data_stored":147819474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":595390160},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,11,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715208,"kb_used_data":142556,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93656632,"statfs":{"total":96636764160,"available":95904391168,"internally_reserved":0,"allocated":145977344,"data_stored":145804242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051682,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":582396,"kb_used_data":145360,"kb_used_omap":4,"kb_used_meta":436987,"kb_avail":93789444,"statfs":{"total":96636764160,"available":96040390656,"internally_reserved":0,"allocated":148848640,"data_stored":148675538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":447475407},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736428,"kb_used_data":145308,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93635412,"statfs":{"total":96636764160,"available":95882661888,"internally_reserved":0,"allocated":148795392,"data_stored":148622290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147992576,"data_stored":147992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":148496384,"data_stored":148496384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145670144,"data_stored":145670144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147034112,"data_stored":147034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:11.669 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:12.245 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:12.245 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:12.272 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-15T21:12:12.137983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293454370,"num_objects":35827,"num_object_clones":0,"num_object_copies":143308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149251,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761620,"kb_used_data":573816,"kb_used_omap":14,"kb_used_meta":2187697,"kb_avail":374725740,"statfs":{"total":386547056640,"available":383719157760,"internally_reserved":0,"allocated":587587584,"data_stored":586895176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2240202290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,12,17,8,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16760832,"num_objects":-2046,"num_object_clones":0,"num_object_copies":-8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004866"},"pg_stats":[{"pgid":"1.f","version":"35'10135","reported_seq":10186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.392463+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:08.392463+0000","last_peered":"2024-09-15T21:12:08.392463+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:08.392463+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10135,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10362","reported_seq":15255,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:11.832049+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:11.832049+0000","last_peered":"2024-09-15T21:12:11.832049+0000","last_clean":"2024-09-15T21:12:11.832049+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:11.832049+0000","last_undegraded":"2024-09-15T21:12:11.832049+0000","last_fullsized":"2024-09-15T21:12:11.832049+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17924096,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10183","reported_seq":10269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.099856+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:10.099856+0000","last_peered":"2024-09-15T21:12:10.099856+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:10.099856+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18422430,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10182,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10260","reported_seq":10335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.140103+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:07.140103+0000","last_peered":"2024-09-15T21:12:07.140103+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:07.140103+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986828,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10393","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.495294+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:08.495294+0000","last_peered":"2024-09-15T21:12:08.495294+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:08.495294+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18560268,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10183","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.320046+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:08.320046+0000","last_peered":"2024-09-15T21:12:08.320046+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:08.320046+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17965056,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10434","reported_seq":10510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.103323+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:07.103323+0000","last_peered":"2024-09-15T21:12:07.103323+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:07.103323+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19150092,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10432,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10529","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.073889+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:07.073889+0000","last_peered":"2024-09-15T21:12:07.073889+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:07.073889+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18281740,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10226","reported_seq":10292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.461144+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:08.461144+0000","last_peered":"2024-09-15T21:12:08.461144+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:08.461144+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18229784,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10335","reported_seq":10422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.038126+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:10.038126+0000","last_peered":"2024-09-15T21:12:10.038126+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:10.038126+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240780,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10145","reported_seq":14812,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:08.482113+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:08.482113+0000","last_peered":"2024-09-15T21:12:08.482113+0000","last_clean":"2024-09-15T21:12:08.482113+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:08.482113+0000","last_undegraded":"2024-09-15T21:12:08.482113+0000","last_fullsized":"2024-09-15T21:12:08.482113+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18145280,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10193","reported_seq":10238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.506605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:08.506605+0000","last_peered":"2024-09-15T21:12:08.506605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:08.506605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18299526,"num_objects":2234,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10192,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10254","reported_seq":12179,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.010755+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:10.010755+0000","last_peered":"2024-09-15T21:12:10.010755+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:10.010755+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18406022,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10253,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10218","reported_seq":10281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.306727+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:08.306727+0000","last_peered":"2024-09-15T21:12:08.306727+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:08.306727+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134284,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10347","reported_seq":10433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:09.976769+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:09.976769+0000","last_peered":"2024-09-15T21:12:09.976769+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:09.976769+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117324,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":53}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10299","reported_seq":10374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.273094+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:07.273094+0000","last_peered":"2024-09-15T21:12:07.273094+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:07.273094+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10297,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293454370,"num_objects":35827,"num_object_clones":0,"num_object_copies":143308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149251,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585023488,"data_stored":585023488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725980,"kb_used_data":144524,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93645860,"statfs":{"total":96636764160,"available":95893360640,"internally_reserved":0,"allocated":147992576,"data_stored":147819474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":595390160},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,11,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715208,"kb_used_data":142556,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93656632,"statfs":{"total":96636764160,"available":95904391168,"internally_reserved":0,"allocated":145977344,"data_stored":145804242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584004,"kb_used_data":141428,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93787836,"statfs":{"total":96636764160,"available":96038744064,"internally_reserved":0,"allocated":144822272,"data_stored":144649170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":453177039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736428,"kb_used_data":145308,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93635412,"statfs":{"total":96636764160,"available":95882661888,"internally_reserved":0,"allocated":148795392,"data_stored":148622290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147992576,"data_stored":147992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144326656,"data_stored":144326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145670144,"data_stored":145670144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147034112,"data_stored":147034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:12.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:12.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10135", 'reported_seq': 10186, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.392463+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:08.392463+0000', 'last_peered': '2024-09-15T21:12:08.392463+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:08.392463+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18161664, 'num_objects': 2217, 'num_object_clones': 0, 'num_object_copies': 8868, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2217, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10135, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -59}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10183", 'reported_seq': 10269, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:10.099856+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:10.099856+0000', 'last_peered': '2024-09-15T21:12:10.099856+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:10.099856+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18422430, 'num_objects': 2250, 'num_object_clones': 0, 'num_object_copies': 9000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2250, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10182, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -8}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10260", 'reported_seq': 10335, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:07.140103+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:07.140103+0000', 'last_peered': '2024-09-15T21:12:07.140103+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:07.140103+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17986828, 'num_objects': 2196, 'num_object_clones': 0, 'num_object_copies': 8784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10258, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10393", 'reported_seq': 10455, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.495294+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:08.495294+0000', 'last_peered': '2024-09-15T21:12:08.495294+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:08.495294+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18560268, 'num_objects': 2266, 'num_object_clones': 0, 'num_object_copies': 9064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2266, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10391, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -45}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10183", 'reported_seq': 10233, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.320046+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:08.320046+0000', 'last_peered': '2024-09-15T21:12:08.320046+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:08.320046+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17965056, 'num_objects': 2193, 'num_object_clones': 0, 'num_object_copies': 8772, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2193, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10183, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:12.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10434", 'reported_seq': 10510, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:07.103323+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:07.103323+0000', 'last_peered': '2024-09-15T21:12:07.103323+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:07.103323+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19150092, 'num_objects': 2338, 'num_object_clones': 0, 'num_object_copies': 9352, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2338, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10432, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -28}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10529", 'reported_seq': 10604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:07.073889+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:07.073889+0000', 'last_peered': '2024-09-15T21:12:07.073889+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:07.073889+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18281740, 'num_objects': 2232, 'num_object_clones': 0, 'num_object_copies': 8928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10527, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -242}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10226", 'reported_seq': 10292, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.461144+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:08.461144+0000', 'last_peered': '2024-09-15T21:12:08.461144+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:08.461144+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18229784, 'num_objects': 2226, 'num_object_clones': 0, 'num_object_copies': 8904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10223, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -106}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10335", 'reported_seq': 10422, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:10.038126+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:10.038126+0000', 'last_peered': '2024-09-15T21:12:10.038126+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:10.038126+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18240780, 'num_objects': 2227, 'num_object_clones': 0, 'num_object_copies': 8908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10333, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -108}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10193", 'reported_seq': 10238, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.506605+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:08.506605+0000', 'last_peered': '2024-09-15T21:12:08.506605+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:08.506605+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'100", 'ondisk_log_start': "27'100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18299526, 'num_objects': 2234, 'num_object_clones': 0, 'num_object_copies': 8936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2234, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10192, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -22}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:12.275 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10254", 'reported_seq': 12179, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:10.010755+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:10.010755+0000', 'last_peered': '2024-09-15T21:12:10.010755+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:10.010755+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18406022, 'num_objects': 2247, 'num_object_clones': 0, 'num_object_copies': 8988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1367, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2247, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10253, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 921, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 880}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10218", 'reported_seq': 10281, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:08.306727+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:08.306727+0000', 'last_peered': '2024-09-15T21:12:08.306727+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:08.306727+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18134284, 'num_objects': 2214, 'num_object_clones': 0, 'num_object_copies': 8856, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2214, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10216, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10347", 'reported_seq': 10433, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:09.976769+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:09.976769+0000', 'last_peered': '2024-09-15T21:12:09.976769+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:09.976769+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19117324, 'num_objects': 2334, 'num_object_clones': 0, 'num_object_copies': 9336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2334, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10345, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 53}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:12.276 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10299", 'reported_seq': 10374, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:07.273094+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:07.273094+0000', 'last_peered': '2024-09-15T21:12:07.273094+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:07.273094+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18429196, 'num_objects': 2250, 'num_object_clones': 0, 'num_object_copies': 9000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2250, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10297, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -65}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:12.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:13.054 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:13.054 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31521/143308 objects degraded (21.995%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":598,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":35827,"data_bytes":293454370,"bytes_used":2827898880,"bytes_avail":383719157760,"bytes_total":386547056640,"degraded_objects":31521,"degraded_total":143308,"degraded_ratio":0.2199528288720797,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":170},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:13.084 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:13.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:13.721 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:13.722 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:13.741 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-09-15T21:12:12.137983+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":293454370,"num_objects":35827,"num_object_clones":0,"num_object_copies":143308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149251,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2761620,"kb_used_data":573816,"kb_used_omap":14,"kb_used_meta":2187697,"kb_avail":374725740,"statfs":{"total":386547056640,"available":383719157760,"internally_reserved":0,"allocated":587587584,"data_stored":586895176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2240202290},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,8,12,17,8,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16760832,"num_objects":-2046,"num_object_clones":0,"num_object_copies":-8184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004866"},"pg_stats":[{"pgid":"1.f","version":"35'10135","reported_seq":10186,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.392463+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:08.392463+0000","last_peered":"2024-09-15T21:12:08.392463+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:08.392463+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18161664,"num_objects":2217,"num_object_clones":0,"num_object_copies":8868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10135,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10362","reported_seq":15255,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:11.832049+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:11.832049+0000","last_peered":"2024-09-15T21:12:11.832049+0000","last_clean":"2024-09-15T21:12:11.832049+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:11.832049+0000","last_undegraded":"2024-09-15T21:12:11.832049+0000","last_fullsized":"2024-09-15T21:12:11.832049+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17924096,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10183","reported_seq":10269,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.099856+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:10.099856+0000","last_peered":"2024-09-15T21:12:10.099856+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:10.099856+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18422430,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10182,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-8}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10260","reported_seq":10335,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.140103+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:07.140103+0000","last_peered":"2024-09-15T21:12:07.140103+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:07.140103+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17986828,"num_objects":2196,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10258,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10393","reported_seq":10455,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.495294+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:08.495294+0000","last_peered":"2024-09-15T21:12:08.495294+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:08.495294+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18560268,"num_objects":2266,"num_object_clones":0,"num_object_copies":9064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10391,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-45}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10183","reported_seq":10233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.320046+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:08.320046+0000","last_peered":"2024-09-15T21:12:08.320046+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:08.320046+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17965056,"num_objects":2193,"num_object_clones":0,"num_object_copies":8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10183,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10434","reported_seq":10510,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.103323+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:07.103323+0000","last_peered":"2024-09-15T21:12:07.103323+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:07.103323+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19150092,"num_objects":2338,"num_object_clones":0,"num_object_copies":9352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10432,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-28}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10529","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.073889+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:07.073889+0000","last_peered":"2024-09-15T21:12:07.073889+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:07.073889+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18281740,"num_objects":2232,"num_object_clones":0,"num_object_copies":8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-242}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10226","reported_seq":10292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.461144+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:08.461144+0000","last_peered":"2024-09-15T21:12:08.461144+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:08.461144+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18229784,"num_objects":2226,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10223,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10335","reported_seq":10422,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.038126+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:10.038126+0000","last_peered":"2024-09-15T21:12:10.038126+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:10.038126+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18240780,"num_objects":2227,"num_object_clones":0,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10333,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-108}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10145","reported_seq":14812,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:08.482113+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:08.482113+0000","last_peered":"2024-09-15T21:12:08.482113+0000","last_clean":"2024-09-15T21:12:08.482113+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:08.482113+0000","last_undegraded":"2024-09-15T21:12:08.482113+0000","last_fullsized":"2024-09-15T21:12:08.482113+0000","mapping_epoch":32,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18145280,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4256,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10193","reported_seq":10238,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.506605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:08.506605+0000","last_peered":"2024-09-15T21:12:08.506605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:08.506605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'100","ondisk_log_start":"27'100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18299526,"num_objects":2234,"num_object_clones":0,"num_object_copies":8936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10192,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-22}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10254","reported_seq":12179,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:10.010755+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:10.010755+0000","last_peered":"2024-09-15T21:12:10.010755+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:10.010755+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18406022,"num_objects":2247,"num_object_clones":0,"num_object_copies":8988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10253,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":880}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10218","reported_seq":10281,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:08.306727+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:08.306727+0000","last_peered":"2024-09-15T21:12:08.306727+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:08.306727+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18134284,"num_objects":2214,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10216,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10347","reported_seq":10433,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:09.976769+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:09.976769+0000","last_peered":"2024-09-15T21:12:09.976769+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:09.976769+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19117324,"num_objects":2334,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10345,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":53}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10299","reported_seq":10374,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:07.273094+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:07.273094+0000","last_peered":"2024-09-15T21:12:07.273094+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:07.273094+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18429196,"num_objects":2250,"num_object_clones":0,"num_object_copies":9000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10297,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-65}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":293454370,"num_objects":35827,"num_object_clones":0,"num_object_copies":143308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31521,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35827,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149251,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":585023488,"data_stored":585023488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160896,"ondisk_log_size":160896,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725980,"kb_used_data":144524,"kb_used_omap":4,"kb_used_meta":581435,"kb_avail":93645860,"statfs":{"total":96636764160,"available":95893360640,"internally_reserved":0,"allocated":147992576,"data_stored":147819474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":595390160},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,11,3,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715208,"kb_used_data":142556,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93656632,"statfs":{"total":96636764160,"available":95904391168,"internally_reserved":0,"allocated":145977344,"data_stored":145804242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,0,2,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584004,"kb_used_data":141428,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93787836,"statfs":{"total":96636764160,"available":96038744064,"internally_reserved":0,"allocated":144822272,"data_stored":144649170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":453177039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247920,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736428,"kb_used_data":145308,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93635412,"statfs":{"total":96636764160,"available":95882661888,"internally_reserved":0,"allocated":148795392,"data_stored":148622290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":605286094},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,1,1,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":147992576,"data_stored":147992576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144326656,"data_stored":144326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":145670144,"data_stored":145670144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":147034112,"data_stored":147034112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:16.743 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:17.364 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:17.365 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:17.396 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-15T21:12:16.139616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286130722,"num_objects":34933,"num_object_clones":0,"num_object_copies":139732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150145,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763784,"kb_used_data":562108,"kb_used_omap":14,"kb_used_meta":2201585,"kb_avail":374723576,"statfs":{"total":386547056640,"available":383716941824,"internally_reserved":0,"allocated":575598592,"data_stored":574906184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2254423602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,9,16,14,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18120704,"num_objects":-2212,"num_object_clones":0,"num_object_copies":-8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"35'10201","reported_seq":10252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.233558+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:13.233558+0000","last_peered":"2024-09-15T21:12:13.233558+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:13.233558+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10201,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10362","reported_seq":15255,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:11.832049+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:11.832049+0000","last_peered":"2024-09-15T21:12:11.832049+0000","last_clean":"2024-09-15T21:12:11.832049+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:11.832049+0000","last_undegraded":"2024-09-15T21:12:11.832049+0000","last_fullsized":"2024-09-15T21:12:11.832049+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17924096,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10224","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.862552+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:14.862552+0000","last_peered":"2024-09-15T21:12:14.862552+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:14.862552+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18086558,"num_objects":2209,"num_object_clones":0,"num_object_copies":8836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10223,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-49}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10323","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.320885+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:12.320885+0000","last_peered":"2024-09-15T21:12:12.320885+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:12.320885+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470732,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10321,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10453","reported_seq":10515,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.411704+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:13.411704+0000","last_peered":"2024-09-15T21:12:13.411704+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:13.411704+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068748,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10451,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10240","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.282419+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:13.282419+0000","last_peered":"2024-09-15T21:12:13.282419+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:13.282419+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10503","reported_seq":10579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.246427+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:12.246427+0000","last_peered":"2024-09-15T21:12:12.246427+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:12.246427+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-97}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10576","reported_seq":10651,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.142700+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:12.142700+0000","last_peered":"2024-09-15T21:12:12.142700+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:12.142700+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896716,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10574,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10281","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.175737+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:13.175737+0000","last_peered":"2024-09-15T21:12:13.175737+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:13.175737+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17779224,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10394","reported_seq":10481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.990060+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:14.990060+0000","last_peered":"2024-09-15T21:12:14.990060+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:14.990060+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10210","reported_seq":14879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:13.225330+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:13.225330+0000","last_peered":"2024-09-15T21:12:13.225330+0000","last_clean":"2024-09-15T21:12:13.225330+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:13.225330+0000","last_undegraded":"2024-09-15T21:12:13.225330+0000","last_fullsized":"2024-09-15T21:12:13.225330+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17612800,"num_objects":2150,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10241","reported_seq":10287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.556267+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:13.556267+0000","last_peered":"2024-09-15T21:12:13.556267+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:13.556267+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17906310,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10328","reported_seq":12345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.123325+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:15.123325+0000","last_peered":"2024-09-15T21:12:15.123325+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:15.123325+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17799814,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10280","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.413009+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:13.413009+0000","last_peered":"2024-09-15T21:12:13.413009+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:13.413009+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626380,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-127}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10416","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.008847+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:15.008847+0000","last_peered":"2024-09-15T21:12:15.008847+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:15.008847+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18552076,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-16}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10358","reported_seq":10432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.346577+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:12.346577+0000","last_peered":"2024-09-15T21:12:12.346577+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:12.346577+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17945868,"num_objects":2191,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286130722,"num_objects":34933,"num_object_clones":0,"num_object_copies":139732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150145,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":572903424,"data_stored":572903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709572,"kb_used_data":140596,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93662268,"statfs":{"total":96636764160,"available":95910162432,"internally_reserved":0,"allocated":143970304,"data_stored":143797202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733620,"kb_used_data":138780,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93638220,"statfs":{"total":96636764160,"available":95885537280,"internally_reserved":0,"allocated":142110720,"data_stored":141937618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584004,"kb_used_data":141428,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93787836,"statfs":{"total":96636764160,"available":96038744064,"internally_reserved":0,"allocated":144822272,"data_stored":144649170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":453177039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736588,"kb_used_data":141304,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93635252,"statfs":{"total":96636764160,"available":95882498048,"internally_reserved":0,"allocated":144695296,"data_stored":144522194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143908864,"data_stored":143908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144326656,"data_stored":144326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141668352,"data_stored":141668352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142999552,"data_stored":142999552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:17.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:17.993 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:17.994 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:18.013 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":327,"stamp":"2024-09-15T21:12:16.139616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":286130722,"num_objects":34933,"num_object_clones":0,"num_object_copies":139732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150145,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2763784,"kb_used_data":562108,"kb_used_omap":14,"kb_used_meta":2201585,"kb_avail":374723576,"statfs":{"total":386547056640,"available":383716941824,"internally_reserved":0,"allocated":575598592,"data_stored":574906184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2254423602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,9,16,14,13,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18120704,"num_objects":-2212,"num_object_clones":0,"num_object_copies":-8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"35'10201","reported_seq":10252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.233558+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:13.233558+0000","last_peered":"2024-09-15T21:12:13.233558+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:13.233558+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10201,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10362","reported_seq":15255,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:11.832049+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:11.832049+0000","last_peered":"2024-09-15T21:12:11.832049+0000","last_clean":"2024-09-15T21:12:11.832049+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:11.832049+0000","last_undegraded":"2024-09-15T21:12:11.832049+0000","last_fullsized":"2024-09-15T21:12:11.832049+0000","mapping_epoch":34,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17924096,"num_objects":2188,"num_object_clones":0,"num_object_copies":8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1028,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10224","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.862552+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:14.862552+0000","last_peered":"2024-09-15T21:12:14.862552+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:14.862552+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18086558,"num_objects":2209,"num_object_clones":0,"num_object_copies":8836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10223,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-49}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10323","reported_seq":10398,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.320885+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:12.320885+0000","last_peered":"2024-09-15T21:12:12.320885+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:12.320885+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17470732,"num_objects":2133,"num_object_clones":0,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10321,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-201}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10453","reported_seq":10515,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.411704+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:13.411704+0000","last_peered":"2024-09-15T21:12:13.411704+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:13.411704+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068748,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10451,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10240","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.282419+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:13.282419+0000","last_peered":"2024-09-15T21:12:13.282419+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:13.282419+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10503","reported_seq":10579,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.246427+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:12.246427+0000","last_peered":"2024-09-15T21:12:12.246427+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:12.246427+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18584844,"num_objects":2269,"num_object_clones":0,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10501,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-97}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10576","reported_seq":10651,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.142700+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:12.142700+0000","last_peered":"2024-09-15T21:12:12.142700+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:12.142700+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17896716,"num_objects":2185,"num_object_clones":0,"num_object_copies":8740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10574,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-289}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10281","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.175737+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:13.175737+0000","last_peered":"2024-09-15T21:12:13.175737+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:13.175737+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17779224,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10394","reported_seq":10481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.990060+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:14.990060+0000","last_peered":"2024-09-15T21:12:14.990060+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:14.990060+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10210","reported_seq":14879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:13.225330+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:13.225330+0000","last_peered":"2024-09-15T21:12:13.225330+0000","last_clean":"2024-09-15T21:12:13.225330+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:13.225330+0000","last_undegraded":"2024-09-15T21:12:13.225330+0000","last_fullsized":"2024-09-15T21:12:13.225330+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17612800,"num_objects":2150,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10241","reported_seq":10287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.556267+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:13.556267+0000","last_peered":"2024-09-15T21:12:13.556267+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:13.556267+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17906310,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10328","reported_seq":12345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.123325+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:15.123325+0000","last_peered":"2024-09-15T21:12:15.123325+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:15.123325+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17799814,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10280","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.413009+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:13.413009+0000","last_peered":"2024-09-15T21:12:13.413009+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:13.413009+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626380,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-127}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10416","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.008847+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:15.008847+0000","last_peered":"2024-09-15T21:12:15.008847+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:15.008847+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18552076,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-16}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10358","reported_seq":10432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:12.346577+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:12.346577+0000","last_peered":"2024-09-15T21:12:12.346577+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:12.346577+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17945868,"num_objects":2191,"num_object_clones":0,"num_object_copies":8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10356,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":286130722,"num_objects":34933,"num_object_clones":0,"num_object_copies":139732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34933,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150145,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":572903424,"data_stored":572903424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160690,"ondisk_log_size":160690,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709572,"kb_used_data":140596,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93662268,"statfs":{"total":96636764160,"available":95910162432,"internally_reserved":0,"allocated":143970304,"data_stored":143797202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733620,"kb_used_data":138780,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93638220,"statfs":{"total":96636764160,"available":95885537280,"internally_reserved":0,"allocated":142110720,"data_stored":141937618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051683,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584004,"kb_used_data":141428,"kb_used_omap":4,"kb_used_meta":442555,"kb_avail":93787836,"statfs":{"total":96636764160,"available":96038744064,"internally_reserved":0,"allocated":144822272,"data_stored":144649170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":453177039},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,9,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736588,"kb_used_data":141304,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93635252,"statfs":{"total":96636764160,"available":95882498048,"internally_reserved":0,"allocated":144695296,"data_stored":144522194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":609545934},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,7,6,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":143908864,"data_stored":143908864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144326656,"data_stored":144326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141668352,"data_stored":141668352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142999552,"data_stored":142999552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:18.014 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10201", 'reported_seq': 10252, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.233558+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:13.233558+0000', 'last_peered': '2024-09-15T21:12:13.233558+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:13.233558+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17620992, 'num_objects': 2151, 'num_object_clones': 0, 'num_object_copies': 8604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2151, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10201, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -125}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10224", 'reported_seq': 10310, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:14.862552+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:14.862552+0000', 'last_peered': '2024-09-15T21:12:14.862552+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:14.862552+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18086558, 'num_objects': 2209, 'num_object_clones': 0, 'num_object_copies': 8836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2209, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10223, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -49}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10323", 'reported_seq': 10398, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:12.320885+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:12.320885+0000', 'last_peered': '2024-09-15T21:12:12.320885+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:12.320885+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17470732, 'num_objects': 2133, 'num_object_clones': 0, 'num_object_copies': 8532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2133, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10321, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -201}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10453", 'reported_seq': 10515, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.411704+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:13.411704+0000', 'last_peered': '2024-09-15T21:12:13.411704+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:13.411704+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18068748, 'num_objects': 2206, 'num_object_clones': 0, 'num_object_copies': 8824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10451, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10240", 'reported_seq': 10290, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.282419+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:13.282419+0000', 'last_peered': '2024-09-15T21:12:13.282419+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:13.282419+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17498112, 'num_objects': 2136, 'num_object_clones': 0, 'num_object_copies': 8544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10240, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -180}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.015 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10503", 'reported_seq': 10579, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:12.246427+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:12.246427+0000', 'last_peered': '2024-09-15T21:12:12.246427+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:12.246427+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18584844, 'num_objects': 2269, 'num_object_clones': 0, 'num_object_copies': 9076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10501, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -97}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10576", 'reported_seq': 10651, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:12.142700+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:12.142700+0000', 'last_peered': '2024-09-15T21:12:12.142700+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:12.142700+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17896716, 'num_objects': 2185, 'num_object_clones': 0, 'num_object_copies': 8740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2185, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10574, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -289}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10281", 'reported_seq': 10347, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.175737+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:13.175737+0000', 'last_peered': '2024-09-15T21:12:13.175737+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:13.175737+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17779224, 'num_objects': 2171, 'num_object_clones': 0, 'num_object_copies': 8684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10278, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10394", 'reported_seq': 10481, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:14.990060+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:14.990060+0000', 'last_peered': '2024-09-15T21:12:14.990060+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:14.990060+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17757452, 'num_objects': 2168, 'num_object_clones': 0, 'num_object_copies': 8672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10392, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -167}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10241", 'reported_seq': 10287, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.556267+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:13.556267+0000', 'last_peered': '2024-09-15T21:12:13.556267+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:13.556267+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17906310, 'num_objects': 2186, 'num_object_clones': 0, 'num_object_copies': 8744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10240, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -70}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:18.016 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10328", 'reported_seq': 12345, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:15.123325+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:15.123325+0000', 'last_peered': '2024-09-15T21:12:15.123325+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:15.123325+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17799814, 'num_objects': 2173, 'num_object_clones': 0, 'num_object_copies': 8692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1322, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2173, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10327, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 966, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 851}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10280", 'reported_seq': 10343, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:13.413009+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:13.413009+0000', 'last_peered': '2024-09-15T21:12:13.413009+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:13.413009+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17626380, 'num_objects': 2152, 'num_object_clones': 0, 'num_object_copies': 8608, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2152, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10278, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -127}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10416", 'reported_seq': 10505, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:15.008847+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:15.008847+0000', 'last_peered': '2024-09-15T21:12:15.008847+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:15.008847+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18552076, 'num_objects': 2265, 'num_object_clones': 0, 'num_object_copies': 9060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2265, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10414, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -16}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:18.017 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10358", 'reported_seq': 10432, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:12.346577+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:12.346577+0000', 'last_peered': '2024-09-15T21:12:12.346577+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:12.346577+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17945868, 'num_objects': 2191, 'num_object_clones': 0, 'num_object_copies': 8764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2191, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10356, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:18.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:18.769 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:18.770 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31476/138508 objects degraded (22.725%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":604,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":34627,"data_bytes":283623970,"bytes_used":2830684160,"bytes_avail":383716372480,"bytes_total":386547056640,"degraded_objects":31476,"degraded_total":138508,"degraded_ratio":0.22725041152857595,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":188},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:18.793 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:18.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:19.371 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:19.371 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:19.406 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-15T21:12:18.140497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":283623970,"num_objects":34627,"num_object_clones":0,"num_object_copies":138508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150451,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2764340,"kb_used_data":554316,"kb_used_omap":14,"kb_used_meta":2209905,"kb_avail":374723020,"statfs":{"total":386547056640,"available":383716372480,"internally_reserved":0,"allocated":567619584,"data_stored":566927176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2262943282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,10,17,18,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005169"},"pg_stats":[{"pgid":"1.f","version":"35'10201","reported_seq":10252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.233558+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:13.233558+0000","last_peered":"2024-09-15T21:12:13.233558+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:13.233558+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17620992,"num_objects":2151,"num_object_clones":0,"num_object_copies":8604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10201,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-125}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10424","reported_seq":15317,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:16.730071+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:16.730071+0000","last_peered":"2024-09-15T21:12:16.730071+0000","last_clean":"2024-09-15T21:12:16.730071+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:16.730071+0000","last_undegraded":"2024-09-15T21:12:16.730071+0000","last_fullsized":"2024-09-15T21:12:16.730071+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17416192,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1090,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10224","reported_seq":10310,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.862552+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:14.862552+0000","last_peered":"2024-09-15T21:12:14.862552+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:14.862552+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18086558,"num_objects":2209,"num_object_clones":0,"num_object_copies":8836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10223,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-49}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10391","reported_seq":10466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.211194+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:17.211194+0000","last_peered":"2024-09-15T21:12:17.211194+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:17.211194+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16913676,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10453","reported_seq":10515,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.411704+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:13.411704+0000","last_peered":"2024-09-15T21:12:13.411704+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:13.411704+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18068748,"num_objects":2206,"num_object_clones":0,"num_object_copies":8824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10451,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10240","reported_seq":10290,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.282419+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:13.282419+0000","last_peered":"2024-09-15T21:12:13.282419+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:13.282419+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17498112,"num_objects":2136,"num_object_clones":0,"num_object_copies":8544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10569","reported_seq":10645,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.034095+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:17.034095+0000","last_peered":"2024-09-15T21:12:17.034095+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:17.034095+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044172,"num_objects":2203,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10635","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.162626+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:17.162626+0000","last_peered":"2024-09-15T21:12:17.162626+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:17.162626+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413388,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10281","reported_seq":10347,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.175737+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:13.175737+0000","last_peered":"2024-09-15T21:12:13.175737+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:13.175737+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17779224,"num_objects":2171,"num_object_clones":0,"num_object_copies":8684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10394","reported_seq":10481,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:14.990060+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:14.990060+0000","last_peered":"2024-09-15T21:12:14.990060+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:14.990060+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17757452,"num_objects":2168,"num_object_clones":0,"num_object_copies":8672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10392,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-167}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10210","reported_seq":14879,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:13.225330+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:13.225330+0000","last_peered":"2024-09-15T21:12:13.225330+0000","last_clean":"2024-09-15T21:12:13.225330+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:13.225330+0000","last_undegraded":"2024-09-15T21:12:13.225330+0000","last_fullsized":"2024-09-15T21:12:13.225330+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17612800,"num_objects":2150,"num_object_clones":0,"num_object_copies":8600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10241","reported_seq":10287,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.556267+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:13.556267+0000","last_peered":"2024-09-15T21:12:13.556267+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:13.556267+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17906310,"num_objects":2186,"num_object_clones":0,"num_object_copies":8744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10240,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-70}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10328","reported_seq":12345,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.123325+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:15.123325+0000","last_peered":"2024-09-15T21:12:15.123325+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:15.123325+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17799814,"num_objects":2173,"num_object_clones":0,"num_object_copies":8692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10327,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":966,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":851}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10280","reported_seq":10343,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:13.413009+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:13.413009+0000","last_peered":"2024-09-15T21:12:13.413009+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:13.413009+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17626380,"num_objects":2152,"num_object_clones":0,"num_object_copies":8608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10278,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-127}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10416","reported_seq":10505,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:15.008847+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:15.008847+0000","last_peered":"2024-09-15T21:12:15.008847+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:15.008847+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18552076,"num_objects":2265,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10414,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-16}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10409","reported_seq":10484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.048719+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:17.048719+0000","last_peered":"2024-09-15T21:12:17.048719+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:17.048719+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528076,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":283623970,"num_objects":34627,"num_object_clones":0,"num_object_copies":138508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34627,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150451,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":987,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":564834304,"data_stored":564834304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160696,"ondisk_log_size":160696,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":709572,"kb_used_data":140596,"kb_used_omap":4,"kb_used_meta":568955,"kb_avail":93662268,"statfs":{"total":96636764160,"available":95910162432,"internally_reserved":0,"allocated":143970304,"data_stored":143797202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":582610640},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,5,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":733620,"kb_used_data":138780,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93638220,"statfs":{"total":96636764160,"available":95885537280,"internally_reserved":0,"allocated":142110720,"data_stored":141937618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,6,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051684,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":584300,"kb_used_data":137564,"kb_used_omap":4,"kb_used_meta":446715,"kb_avail":93787540,"statfs":{"total":96636764160,"available":96038440960,"internally_reserved":0,"allocated":140865536,"data_stored":140692434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":457436879},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,7,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736848,"kb_used_data":137376,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93634992,"statfs":{"total":96636764160,"available":95882231808,"internally_reserved":0,"allocated":140673024,"data_stored":140499922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139898880,"data_stored":139898880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":140267520,"data_stored":140267520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":141668352,"data_stored":141668352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":142999552,"data_stored":142999552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:22.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:23.023 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:23.024 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:23.061 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-15T21:12:22.142291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277619234,"num_objects":33894,"num_object_clones":0,"num_object_copies":135576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151184,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801320,"kb_used_data":542712,"kb_used_omap":14,"kb_used_meta":2258481,"kb_avail":374686040,"statfs":{"total":386547056640,"available":383678504960,"internally_reserved":0,"allocated":555737088,"data_stored":555044680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2312685106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,11,17,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16416768,"num_objects":-2004,"num_object_clones":0,"num_object_copies":-8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005261"},"pg_stats":[{"pgid":"1.f","version":"35'10272","reported_seq":10321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.492666+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:18.492666+0000","last_peered":"2024-09-15T21:12:18.492666+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:18.492666+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10471","reported_seq":15363,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:22.022573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:22.022573+0000","last_peered":"2024-09-15T21:12:22.022573+0000","last_clean":"2024-09-15T21:12:22.022573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:22.022573+0000","last_undegraded":"2024-09-15T21:12:22.022573+0000","last_fullsized":"2024-09-15T21:12:22.022573+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17031168,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.062883+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:20.062883+0000","last_peered":"2024-09-15T21:12:20.062883+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:20.062883+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570462,"num_objects":2146,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10391","reported_seq":10466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.211194+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:17.211194+0000","last_peered":"2024-09-15T21:12:17.211194+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:17.211194+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16913676,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10528","reported_seq":10590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.617364+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:18.617364+0000","last_peered":"2024-09-15T21:12:18.617364+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:18.617364+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454348,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10306","reported_seq":10357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.725978+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:18.725978+0000","last_peered":"2024-09-15T21:12:18.725978+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:18.725978+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16957440,"num_objects":2070,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10569","reported_seq":10645,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.034095+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:17.034095+0000","last_peered":"2024-09-15T21:12:17.034095+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:17.034095+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044172,"num_objects":2203,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10635","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.162626+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:17.162626+0000","last_peered":"2024-09-15T21:12:17.162626+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:17.162626+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413388,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10349","reported_seq":10414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.337164+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:18.337164+0000","last_peered":"2024-09-15T21:12:18.337164+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:18.337164+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17222168,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10439","reported_seq":10526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.059286+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:20.059286+0000","last_peered":"2024-09-15T21:12:20.059286+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:20.059286+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-212}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10273","reported_seq":14942,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:18.697417+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:18.697417+0000","last_peered":"2024-09-15T21:12:18.697417+0000","last_clean":"2024-09-15T21:12:18.697417+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:18.697417+0000","last_undegraded":"2024-09-15T21:12:18.697417+0000","last_fullsized":"2024-09-15T21:12:18.697417+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17096704,"num_objects":2087,"num_object_clones":0,"num_object_copies":8348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4384,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10299","reported_seq":10344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.728291+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:18.728291+0000","last_peered":"2024-09-15T21:12:18.728291+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:18.728291+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17431174,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10379","reported_seq":12504,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.135153+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:20.135153+0000","last_peered":"2024-09-15T21:12:20.135153+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:20.135153+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17382022,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10356","reported_seq":10419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.498582+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:18.498582+0000","last_peered":"2024-09-15T21:12:18.498582+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:18.498582+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17003788,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-203}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10466","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.175132+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:20.175132+0000","last_peered":"2024-09-15T21:12:20.175132+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:20.175132+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18142476,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-66}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10409","reported_seq":10484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.048719+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:17.048719+0000","last_peered":"2024-09-15T21:12:17.048719+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:17.048719+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528076,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":277619234,"num_objects":33894,"num_object_clones":0,"num_object_copies":135576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151184,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552919040,"data_stored":552919040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727788,"kb_used_data":136600,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93644052,"statfs":{"total":96636764160,"available":95891509248,"internally_reserved":0,"allocated":139878400,"data_stored":139705298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734012,"kb_used_data":135020,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93637828,"statfs":{"total":96636764160,"available":95885135872,"internally_reserved":0,"allocated":138260480,"data_stored":138087378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602672,"kb_used_data":133716,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93769168,"statfs":{"total":96636764160,"available":96019628032,"internally_reserved":0,"allocated":136925184,"data_stored":136752082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736848,"kb_used_data":137376,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93634992,"statfs":{"total":96636764160,"available":95882231808,"internally_reserved":0,"allocated":140673024,"data_stored":140499922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139898880,"data_stored":139898880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136310784,"data_stored":136310784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137781248,"data_stored":137781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138928128,"data_stored":138928128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:23.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:23.665 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:23.769 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:23.771 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-15T21:12:22.142291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277619234,"num_objects":33894,"num_object_clones":0,"num_object_copies":135576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151184,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801320,"kb_used_data":542712,"kb_used_omap":14,"kb_used_meta":2258481,"kb_avail":374686040,"statfs":{"total":386547056640,"available":383678504960,"internally_reserved":0,"allocated":555737088,"data_stored":555044680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2312685106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,11,17,16,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16416768,"num_objects":-2004,"num_object_clones":0,"num_object_copies":-8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2004,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005261"},"pg_stats":[{"pgid":"1.f","version":"35'10272","reported_seq":10321,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.492666+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:18.492666+0000","last_peered":"2024-09-15T21:12:18.492666+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:18.492666+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17039360,"num_objects":2080,"num_object_clones":0,"num_object_copies":8320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10272,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10471","reported_seq":15363,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:22.022573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:22.022573+0000","last_peered":"2024-09-15T21:12:22.022573+0000","last_clean":"2024-09-15T21:12:22.022573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:22.022573+0000","last_undegraded":"2024-09-15T21:12:22.022573+0000","last_fullsized":"2024-09-15T21:12:22.022573+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17031168,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.062883+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:20.062883+0000","last_peered":"2024-09-15T21:12:20.062883+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:20.062883+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570462,"num_objects":2146,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10391","reported_seq":10466,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.211194+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:17.211194+0000","last_peered":"2024-09-15T21:12:17.211194+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:17.211194+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16913676,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10389,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-269}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10528","reported_seq":10590,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.617364+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:18.617364+0000","last_peered":"2024-09-15T21:12:18.617364+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:18.617364+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17454348,"num_objects":2131,"num_object_clones":0,"num_object_copies":8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-180}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10306","reported_seq":10357,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.725978+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:18.725978+0000","last_peered":"2024-09-15T21:12:18.725978+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:18.725978+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16957440,"num_objects":2070,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10306,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10569","reported_seq":10645,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.034095+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:17.034095+0000","last_peered":"2024-09-15T21:12:17.034095+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:17.034095+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18044172,"num_objects":2203,"num_object_clones":0,"num_object_copies":8812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-163}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10635","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.162626+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:17.162626+0000","last_peered":"2024-09-15T21:12:17.162626+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:17.162626+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17413388,"num_objects":2126,"num_object_clones":0,"num_object_copies":8504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10633,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-348}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10349","reported_seq":10414,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.337164+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:18.337164+0000","last_peered":"2024-09-15T21:12:18.337164+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:18.337164+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17222168,"num_objects":2103,"num_object_clones":0,"num_object_copies":8412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10346,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-229}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10439","reported_seq":10526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.059286+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:20.059286+0000","last_peered":"2024-09-15T21:12:20.059286+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:20.059286+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-212}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10273","reported_seq":14942,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:18.697417+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:18.697417+0000","last_peered":"2024-09-15T21:12:18.697417+0000","last_clean":"2024-09-15T21:12:18.697417+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:18.697417+0000","last_undegraded":"2024-09-15T21:12:18.697417+0000","last_fullsized":"2024-09-15T21:12:18.697417+0000","mapping_epoch":32,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17096704,"num_objects":2087,"num_object_clones":0,"num_object_copies":8348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4384,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10299","reported_seq":10344,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.728291+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:18.728291+0000","last_peered":"2024-09-15T21:12:18.728291+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:18.728291+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17431174,"num_objects":2128,"num_object_clones":0,"num_object_copies":8512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10298,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-128}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10379","reported_seq":12504,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.135153+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:20.135153+0000","last_peered":"2024-09-15T21:12:20.135153+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:20.135153+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17382022,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10356","reported_seq":10419,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:18.498582+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:18.498582+0000","last_peered":"2024-09-15T21:12:18.498582+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:18.498582+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17003788,"num_objects":2076,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10354,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-203}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10466","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.175132+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:20.175132+0000","last_peered":"2024-09-15T21:12:20.175132+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:20.175132+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18142476,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-66}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10409","reported_seq":10484,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:17.048719+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:17.048719+0000","last_peered":"2024-09-15T21:12:17.048719+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:17.048719+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528076,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-175}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":277619234,"num_objects":33894,"num_object_clones":0,"num_object_copies":135576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33894,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151184,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552919040,"data_stored":552919040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160929,"ondisk_log_size":160929,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727788,"kb_used_data":136600,"kb_used_omap":4,"kb_used_meta":591163,"kb_avail":93644052,"statfs":{"total":96636764160,"available":95891509248,"internally_reserved":0,"allocated":139878400,"data_stored":139705298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":605351632},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,6,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734012,"kb_used_data":135020,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93637828,"statfs":{"total":96636764160,"available":95885135872,"internally_reserved":0,"allocated":138260480,"data_stored":138087378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602672,"kb_used_data":133716,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93769168,"statfs":{"total":96636764160,"available":96019628032,"internally_reserved":0,"allocated":136925184,"data_stored":136752082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247922,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":736848,"kb_used_data":137376,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93634992,"statfs":{"total":96636764160,"available":95882231808,"internally_reserved":0,"allocated":140673024,"data_stored":140499922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":613805774},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139898880,"data_stored":139898880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136310784,"data_stored":136310784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137781248,"data_stored":137781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":138928128,"data_stored":138928128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:23.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:23.772 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10272", 'reported_seq': 10321, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.492666+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:18.492666+0000', 'last_peered': '2024-09-15T21:12:18.492666+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:18.492666+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17039360, 'num_objects': 2080, 'num_object_clones': 0, 'num_object_copies': 8320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2080, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10272, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.772 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:23.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10287", 'reported_seq': 10373, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:20.062883+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:20.062883+0000', 'last_peered': '2024-09-15T21:12:20.062883+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:20.062883+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17570462, 'num_objects': 2146, 'num_object_clones': 0, 'num_object_copies': 8584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2146, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10286, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -112}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:23.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:23.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10391", 'reported_seq': 10466, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:17.211194+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:17.211194+0000', 'last_peered': '2024-09-15T21:12:17.211194+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:17.211194+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16913676, 'num_objects': 2065, 'num_object_clones': 0, 'num_object_copies': 8260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10389, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -269}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:23.773 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:23.773 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10528", 'reported_seq': 10590, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.617364+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:18.617364+0000', 'last_peered': '2024-09-15T21:12:18.617364+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:18.617364+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17454348, 'num_objects': 2131, 'num_object_clones': 0, 'num_object_copies': 8524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2131, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10526, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -180}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10306", 'reported_seq': 10357, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.725978+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:18.725978+0000', 'last_peered': '2024-09-15T21:12:18.725978+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:18.725978+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16957440, 'num_objects': 2070, 'num_object_clones': 0, 'num_object_copies': 8280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2070, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10306, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -246}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10569", 'reported_seq': 10645, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:17.034095+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:17.034095+0000', 'last_peered': '2024-09-15T21:12:17.034095+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:17.034095+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18044172, 'num_objects': 2203, 'num_object_clones': 0, 'num_object_copies': 8812, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2203, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10567, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -163}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:23.774 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10635", 'reported_seq': 10708, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:17.162626+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:17.162626+0000', 'last_peered': '2024-09-15T21:12:17.162626+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:17.162626+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17413388, 'num_objects': 2126, 'num_object_clones': 0, 'num_object_copies': 8504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2126, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10633, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -348}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10349", 'reported_seq': 10414, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.337164+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:18.337164+0000', 'last_peered': '2024-09-15T21:12:18.337164+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:18.337164+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17222168, 'num_objects': 2103, 'num_object_clones': 0, 'num_object_copies': 8412, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2103, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10346, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -229}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10439", 'reported_seq': 10526, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:20.059286+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:20.059286+0000', 'last_peered': '2024-09-15T21:12:20.059286+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:20.059286+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17388812, 'num_objects': 2123, 'num_object_clones': 0, 'num_object_copies': 8492, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2123, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10437, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -212}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:23.775 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10299", 'reported_seq': 10344, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.728291+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:18.728291+0000', 'last_peered': '2024-09-15T21:12:18.728291+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:18.728291+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'200", 'ondisk_log_start': "27'200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17431174, 'num_objects': 2128, 'num_object_clones': 0, 'num_object_copies': 8512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2128, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10298, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -128}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10379", 'reported_seq': 12504, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:20.135153+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:20.135153+0000', 'last_peered': '2024-09-15T21:12:20.135153+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:20.135153+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17382022, 'num_objects': 2122, 'num_object_clones': 0, 'num_object_copies': 8488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1268, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10378, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1020, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 854}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10356", 'reported_seq': 10419, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:18.498582+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:18.498582+0000', 'last_peered': '2024-09-15T21:12:18.498582+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:18.498582+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17003788, 'num_objects': 2076, 'num_object_clones': 0, 'num_object_copies': 8304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2076, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10354, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -203}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:23.776 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10466", 'reported_seq': 10555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:20.175132+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:20.175132+0000', 'last_peered': '2024-09-15T21:12:20.175132+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:20.175132+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 18142476, 'num_objects': 2215, 'num_object_clones': 0, 'num_object_copies': 8860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2215, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10464, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -66}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:23.777 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:23.777 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10409", 'reported_seq': 10484, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:17.048719+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:17.048719+0000', 'last_peered': '2024-09-15T21:12:17.048719+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:17.048719+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17528076, 'num_objects': 2140, 'num_object_clones': 0, 'num_object_copies': 8560, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2140, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10407, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -175}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:23.777 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:24.520 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:24.520 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31422/132700 objects degraded (23.679%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":610,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":33175,"data_bytes":271729186,"bytes_used":2869886976,"bytes_avail":383677169664,"bytes_total":386547056640,"degraded_objects":31422,"degraded_total":132700,"degraded_ratio":0.23678975131876412,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:24.558 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:24.558 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:25.205 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:25.205 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:25.233 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":331,"stamp":"2024-09-15T21:12:24.142927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":271729186,"num_objects":33175,"num_object_clones":0,"num_object_copies":132700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151903,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802624,"kb_used_data":534324,"kb_used_omap":14,"kb_used_meta":2268209,"kb_avail":374684736,"statfs":{"total":386547056640,"available":383677169664,"internally_reserved":0,"allocated":547147776,"data_stored":546455368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2322646578},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,3,12,18,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21725184,"num_objects":-2652,"num_object_clones":0,"num_object_copies":-10608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-99,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2652,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004943"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.711912+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:23.711912+0000","last_peered":"2024-09-15T21:12:23.711912+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:23.711912+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10471","reported_seq":15363,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:22.022573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:22.022573+0000","last_peered":"2024-09-15T21:12:22.022573+0000","last_clean":"2024-09-15T21:12:22.022573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:22.022573+0000","last_undegraded":"2024-09-15T21:12:22.022573+0000","last_fullsized":"2024-09-15T21:12:22.022573+0000","mapping_epoch":34,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17031168,"num_objects":2079,"num_object_clones":0,"num_object_copies":8316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1137,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10287","reported_seq":10373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.062883+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:20.062883+0000","last_peered":"2024-09-15T21:12:20.062883+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:20.062883+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'200","ondisk_log_start":"27'200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17570462,"num_objects":2146,"num_object_clones":0,"num_object_copies":8584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10286,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-112}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10446","reported_seq":10521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:22.104157+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:22.104157+0000","last_peered":"2024-09-15T21:12:22.104157+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:22.104157+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16463116,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10444,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-324}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10603","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.731593+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:23.731593+0000","last_peered":"2024-09-15T21:12:23.731593+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:23.731593+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839948,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10366","reported_seq":10416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.749406+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:23.749406+0000","last_peered":"2024-09-15T21:12:23.749406+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:23.749406+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16465920,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10632","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:22.326321+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:22.326321+0000","last_peered":"2024-09-15T21:12:22.326321+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:22.326321+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17528076,"num_objects":2140,"num_object_clones":0,"num_object_copies":8560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10630,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10691","reported_seq":10766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:21.982046+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:21.982046+0000","last_peered":"2024-09-15T21:12:21.982046+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:21.982046+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16954636,"num_objects":2070,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-404}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10425","reported_seq":10491,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.852648+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:23.852648+0000","last_peered":"2024-09-15T21:12:23.852648+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:23.852648+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599576,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10439","reported_seq":10526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.059286+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:20.059286+0000","last_peered":"2024-09-15T21:12:20.059286+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:20.059286+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17388812,"num_objects":2123,"num_object_clones":0,"num_object_copies":8492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10437,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-212}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10332","reported_seq":15001,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:23.775826+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:23.775826+0000","last_peered":"2024-09-15T21:12:23.775826+0000","last_clean":"2024-09-15T21:12:23.775826+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:23.775826+0000","last_undegraded":"2024-09-15T21:12:23.775826+0000","last_fullsized":"2024-09-15T21:12:23.775826+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16613376,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10362","reported_seq":10408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.658599+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:23.658599+0000","last_peered":"2024-09-15T21:12:23.658599+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:23.658599+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915078,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10379","reported_seq":12504,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.135153+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:20.135153+0000","last_peered":"2024-09-15T21:12:20.135153+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:20.135153+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17382022,"num_objects":2122,"num_object_clones":0,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10378,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1020,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":854}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10425","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.894082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:23.894082+0000","last_peered":"2024-09-15T21:12:23.894082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:23.894082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16438540,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10466","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:20.175132+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:20.175132+0000","last_peered":"2024-09-15T21:12:20.175132+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:20.175132+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18142476,"num_objects":2215,"num_object_clones":0,"num_object_copies":8860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-66}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10483","reported_seq":10557,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:22.377844+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:22.377844+0000","last_peered":"2024-09-15T21:12:22.377844+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:22.377844+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-249}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":271729186,"num_objects":33175,"num_object_clones":0,"num_object_copies":132700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33175,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":151903,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1041,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544833536,"data_stored":544833536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160848,"ondisk_log_size":160848,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727380,"kb_used_data":132036,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93644460,"statfs":{"total":96636764160,"available":95891927040,"internally_reserved":0,"allocated":135204864,"data_stored":135031762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":734012,"kb_used_data":135020,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93637828,"statfs":{"total":96636764160,"available":95885135872,"internally_reserved":0,"allocated":138260480,"data_stored":138087378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051685,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602672,"kb_used_data":133716,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93769168,"statfs":{"total":96636764160,"available":96019628032,"internally_reserved":0,"allocated":136925184,"data_stored":136752082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247923,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738560,"kb_used_data":133552,"kb_used_omap":4,"kb_used_meta":604987,"kb_avail":93633280,"statfs":{"total":96636764160,"available":95880478720,"internally_reserved":0,"allocated":136757248,"data_stored":136584146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":619507406},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135917568,"data_stored":135917568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":136310784,"data_stored":136310784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137781248,"data_stored":137781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134823936,"data_stored":134823936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:28.234 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:28.860 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:28.861 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:28.883 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-15T21:12:28.144511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267362850,"num_objects":32642,"num_object_clones":0,"num_object_copies":130568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152436,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799412,"kb_used_data":522804,"kb_used_omap":14,"kb_used_meta":2276529,"kb_avail":374687948,"statfs":{"total":386547056640,"available":383680458752,"internally_reserved":0,"allocated":535351296,"data_stored":534658888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2331166258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,1,5,8,10,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18767872,"num_objects":-2291,"num_object_clones":0,"num_object_copies":-9164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.711912+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:23.711912+0000","last_peered":"2024-09-15T21:12:23.711912+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:23.711912+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10529","reported_seq":15422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:27.170877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:27.170877+0000","last_peered":"2024-09-15T21:12:27.170877+0000","last_clean":"2024-09-15T21:12:27.170877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:27.170877+0000","last_undegraded":"2024-09-15T21:12:27.170877+0000","last_fullsized":"2024-09-15T21:12:27.170877+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16556032,"num_objects":2021,"num_object_clones":0,"num_object_copies":8084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10344","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.960132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:24.960132+0000","last_peered":"2024-09-15T21:12:24.960132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:24.960132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17103518,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10343,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10500","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.293760+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:27.293760+0000","last_peered":"2024-09-15T21:12:27.293760+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:27.293760+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16020748,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10603","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.731593+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:23.731593+0000","last_peered":"2024-09-15T21:12:23.731593+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:23.731593+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839948,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10366","reported_seq":10416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.749406+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:23.749406+0000","last_peered":"2024-09-15T21:12:23.749406+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:23.749406+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16465920,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10704","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.428659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:27.428659+0000","last_peered":"2024-09-15T21:12:27.428659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:27.428659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16938252,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10750","reported_seq":10825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.337570+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:27.337570+0000","last_peered":"2024-09-15T21:12:27.337570+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:27.337570+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471308,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10425","reported_seq":10491,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.852648+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:23.852648+0000","last_peered":"2024-09-15T21:12:23.852648+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:23.852648+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599576,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10496","reported_seq":10583,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.914664+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:24.914664+0000","last_peered":"2024-09-15T21:12:24.914664+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:24.914664+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10332","reported_seq":15001,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:23.775826+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:23.775826+0000","last_peered":"2024-09-15T21:12:23.775826+0000","last_clean":"2024-09-15T21:12:23.775826+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:23.775826+0000","last_undegraded":"2024-09-15T21:12:23.775826+0000","last_fullsized":"2024-09-15T21:12:23.775826+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16613376,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10362","reported_seq":10408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.658599+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:23.658599+0000","last_peered":"2024-09-15T21:12:23.658599+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:23.658599+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915078,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10432","reported_seq":12671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.265627+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:25.265627+0000","last_peered":"2024-09-15T21:12:25.265627+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:25.265627+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16947846,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10431,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10425","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.894082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:23.894082+0000","last_peered":"2024-09-15T21:12:23.894082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:23.894082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16438540,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10516","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.153286+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:25.153286+0000","last_peered":"2024-09-15T21:12:25.153286+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:25.153286+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732876,"num_objects":2165,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10556","reported_seq":10631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.419282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:27.419282+0000","last_peered":"2024-09-15T21:12:27.419282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:27.419282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16323852,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":267362850,"num_objects":32642,"num_object_clones":0,"num_object_copies":130568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152436,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532717568,"data_stored":532717568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727380,"kb_used_data":132036,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93644460,"statfs":{"total":96636764160,"available":95891927040,"internally_reserved":0,"allocated":135204864,"data_stored":135031762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730132,"kb_used_data":131140,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93641708,"statfs":{"total":96636764160,"available":95889108992,"internally_reserved":0,"allocated":134287360,"data_stored":134114258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602980,"kb_used_data":129876,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93768860,"statfs":{"total":96636764160,"available":96019312640,"internally_reserved":0,"allocated":132993024,"data_stored":132819922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738920,"kb_used_data":129752,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632920,"statfs":{"total":96636764160,"available":95880110080,"internally_reserved":0,"allocated":132866048,"data_stored":132692946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131899392,"data_stored":131899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132186112,"data_stored":132186112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133808128,"data_stored":133808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134823936,"data_stored":134823936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:28.884 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:29.484 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:29.484 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:29.513 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-15T21:12:28.144511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":267362850,"num_objects":32642,"num_object_clones":0,"num_object_copies":130568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152436,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2799412,"kb_used_data":522804,"kb_used_omap":14,"kb_used_meta":2276529,"kb_avail":374687948,"statfs":{"total":386547056640,"available":383680458752,"internally_reserved":0,"allocated":535351296,"data_stored":534658888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2331166258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,7,1,5,8,10,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18767872,"num_objects":-2291,"num_object_clones":0,"num_object_copies":-9164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2291,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004895"},"pg_stats":[{"pgid":"1.f","version":"35'10341","reported_seq":10393,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.711912+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:23.711912+0000","last_peered":"2024-09-15T21:12:23.711912+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:23.711912+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16474112,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10341,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-265}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10529","reported_seq":15422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:27.170877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:27.170877+0000","last_peered":"2024-09-15T21:12:27.170877+0000","last_clean":"2024-09-15T21:12:27.170877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:27.170877+0000","last_undegraded":"2024-09-15T21:12:27.170877+0000","last_fullsized":"2024-09-15T21:12:27.170877+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16556032,"num_objects":2021,"num_object_clones":0,"num_object_copies":8084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10344","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.960132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:24.960132+0000","last_peered":"2024-09-15T21:12:24.960132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:24.960132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17103518,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10343,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10500","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.293760+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:27.293760+0000","last_peered":"2024-09-15T21:12:27.293760+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:27.293760+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16020748,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10603","reported_seq":10665,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.731593+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:23.731593+0000","last_peered":"2024-09-15T21:12:23.731593+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:23.731593+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16839948,"num_objects":2056,"num_object_clones":0,"num_object_copies":8224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10601,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-255}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10366","reported_seq":10416,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.749406+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:23.749406+0000","last_peered":"2024-09-15T21:12:23.749406+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:23.749406+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16465920,"num_objects":2010,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10366,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-306}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10704","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.428659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:27.428659+0000","last_peered":"2024-09-15T21:12:27.428659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:27.428659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16938252,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10750","reported_seq":10825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.337570+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:27.337570+0000","last_peered":"2024-09-15T21:12:27.337570+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:27.337570+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471308,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10425","reported_seq":10491,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.852648+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:23.852648+0000","last_peered":"2024-09-15T21:12:23.852648+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:23.852648+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16599576,"num_objects":2027,"num_object_clones":0,"num_object_copies":8108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10422,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-305}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10496","reported_seq":10583,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.914664+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:24.914664+0000","last_peered":"2024-09-15T21:12:24.914664+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:24.914664+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10332","reported_seq":15001,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:23.775826+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:23.775826+0000","last_peered":"2024-09-15T21:12:23.775826+0000","last_clean":"2024-09-15T21:12:23.775826+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:23.775826+0000","last_undegraded":"2024-09-15T21:12:23.775826+0000","last_fullsized":"2024-09-15T21:12:23.775826+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16613376,"num_objects":2028,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4443,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10362","reported_seq":10408,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.658599+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:23.658599+0000","last_peered":"2024-09-15T21:12:23.658599+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:23.658599+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16915078,"num_objects":2065,"num_object_clones":0,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10361,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-191}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10432","reported_seq":12671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.265627+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:25.265627+0000","last_peered":"2024-09-15T21:12:25.265627+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:25.265627+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16947846,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10431,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10425","reported_seq":10487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:23.894082+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:23.894082+0000","last_peered":"2024-09-15T21:12:23.894082+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:23.894082+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16438540,"num_objects":2007,"num_object_clones":0,"num_object_copies":8028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10423,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10516","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.153286+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:25.153286+0000","last_peered":"2024-09-15T21:12:25.153286+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:25.153286+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732876,"num_objects":2165,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10556","reported_seq":10631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.419282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:27.419282+0000","last_peered":"2024-09-15T21:12:27.419282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:27.419282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16323852,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":267362850,"num_objects":32642,"num_object_clones":0,"num_object_copies":130568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152436,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532717568,"data_stored":532717568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160681,"ondisk_log_size":160681,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727380,"kb_used_data":132036,"kb_used_omap":4,"kb_used_meta":595323,"kb_avail":93644460,"statfs":{"total":96636764160,"available":95891927040,"internally_reserved":0,"allocated":135204864,"data_stored":135031762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":609611472},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,2,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730132,"kb_used_data":131140,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93641708,"statfs":{"total":96636764160,"available":95889108992,"internally_reserved":0,"allocated":134287360,"data_stored":134114258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602980,"kb_used_data":129876,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93768860,"statfs":{"total":96636764160,"available":96019312640,"internally_reserved":0,"allocated":132993024,"data_stored":132819922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738920,"kb_used_data":129752,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632920,"statfs":{"total":96636764160,"available":95880110080,"internally_reserved":0,"allocated":132866048,"data_stored":132692946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131899392,"data_stored":131899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132186112,"data_stored":132186112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133808128,"data_stored":133808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":134823936,"data_stored":134823936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:29.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10341", 'reported_seq': 10393, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.711912+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:23.711912+0000', 'last_peered': '2024-09-15T21:12:23.711912+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:23.711912+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16474112, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10341, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -265}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10344", 'reported_seq': 10430, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:24.960132+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:24.960132+0000', 'last_peered': '2024-09-15T21:12:24.960132+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:24.960132+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17103518, 'num_objects': 2089, 'num_object_clones': 0, 'num_object_copies': 8356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2089, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10343, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -169}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10500", 'reported_seq': 10575, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:27.293760+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:27.293760+0000', 'last_peered': '2024-09-15T21:12:27.293760+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:27.293760+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16020748, 'num_objects': 1956, 'num_object_clones': 0, 'num_object_copies': 7824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1956, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10498, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -378}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:29.515 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10603", 'reported_seq': 10665, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.731593+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:23.731593+0000', 'last_peered': '2024-09-15T21:12:23.731593+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:23.731593+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16839948, 'num_objects': 2056, 'num_object_clones': 0, 'num_object_copies': 8224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2056, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10601, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -255}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10366", 'reported_seq': 10416, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.749406+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:23.749406+0000', 'last_peered': '2024-09-15T21:12:23.749406+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:23.749406+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16465920, 'num_objects': 2010, 'num_object_clones': 0, 'num_object_copies': 8040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2010, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10366, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -306}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10704", 'reported_seq': 10780, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:27.428659+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:27.428659+0000', 'last_peered': '2024-09-15T21:12:27.428659+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:27.428659+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16938252, 'num_objects': 2068, 'num_object_clones': 0, 'num_object_copies': 8272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2068, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10702, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -298}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10750", 'reported_seq': 10825, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:27.337570+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:27.337570+0000', 'last_peered': '2024-09-15T21:12:27.337570+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:27.337570+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16471308, 'num_objects': 2011, 'num_object_clones': 0, 'num_object_copies': 8044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2011, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10748, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10425", 'reported_seq': 10491, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.852648+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:23.852648+0000', 'last_peered': '2024-09-15T21:12:23.852648+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:23.852648+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16599576, 'num_objects': 2027, 'num_object_clones': 0, 'num_object_copies': 8108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2027, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10422, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -305}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.516 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10496", 'reported_seq': 10583, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:24.914664+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:24.914664+0000', 'last_peered': '2024-09-15T21:12:24.914664+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:24.914664+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16921868, 'num_objects': 2066, 'num_object_clones': 0, 'num_object_copies': 8264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2066, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10494, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -269}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10362", 'reported_seq': 10408, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.658599+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:23.658599+0000', 'last_peered': '2024-09-15T21:12:23.658599+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:23.658599+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'300", 'ondisk_log_start': "27'300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16915078, 'num_objects': 2065, 'num_object_clones': 0, 'num_object_copies': 8260, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2065, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10361, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -191}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10432", 'reported_seq': 12671, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:25.265627+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:25.265627+0000', 'last_peered': '2024-09-15T21:12:25.265627+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:25.265627+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16947846, 'num_objects': 2069, 'num_object_clones': 0, 'num_object_copies': 8276, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1211, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2069, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10431, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1077, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10425", 'reported_seq': 10487, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:23.894082+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:23.894082+0000', 'last_peered': '2024-09-15T21:12:23.894082+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:23.894082+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16438540, 'num_objects': 2007, 'num_object_clones': 0, 'num_object_copies': 8028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2007, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10423, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:29.517 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10516", 'reported_seq': 10604, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:25.153286+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:25.153286+0000', 'last_peered': '2024-09-15T21:12:25.153286+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:25.153286+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17732876, 'num_objects': 2165, 'num_object_clones': 0, 'num_object_copies': 8660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10514, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -116}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:29.518 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:29.518 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10556", 'reported_seq': 10631, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:27.419282+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:27.419282+0000', 'last_peered': '2024-09-15T21:12:27.419282+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:27.419282+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16323852, 'num_objects': 1993, 'num_object_clones': 0, 'num_object_copies': 7972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1993, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10554, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -322}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:29.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:30.297 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:30.297 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31365/130568 objects degraded (24.022%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":616,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":32642,"data_bytes":267362850,"bytes_used":2866597888,"bytes_avail":383680458752,"bytes_total":386547056640,"degraded_objects":31365,"degraded_total":130568,"degraded_ratio":0.24021965565835426,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:30.316 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:30.316 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:30.901 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:30.902 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:30.928 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-15T21:12:30.145200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":263725602,"num_objects":32198,"num_object_clones":0,"num_object_copies":128792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152880,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2800256,"kb_used_data":519484,"kb_used_omap":14,"kb_used_meta":2280689,"kb_avail":374687104,"statfs":{"total":386547056640,"available":383679594496,"internally_reserved":0,"allocated":531951616,"data_stored":531259208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2335426098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,3,5,10,6,18,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19898368,"num_objects":-2429,"num_object_clones":0,"num_object_copies":-9716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2429,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004702"},"pg_stats":[{"pgid":"1.f","version":"35'10407","reported_seq":10457,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.967149+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:28.967149+0000","last_peered":"2024-09-15T21:12:28.967149+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:28.967149+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15933440,"num_objects":1945,"num_object_clones":0,"num_object_copies":7780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10407,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10529","reported_seq":15422,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:27.170877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:27.170877+0000","last_peered":"2024-09-15T21:12:27.170877+0000","last_clean":"2024-09-15T21:12:27.170877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:27.170877+0000","last_undegraded":"2024-09-15T21:12:27.170877+0000","last_fullsized":"2024-09-15T21:12:27.170877+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16556032,"num_objects":2021,"num_object_clones":0,"num_object_copies":8084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1195,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10344","reported_seq":10430,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.960132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:24.960132+0000","last_peered":"2024-09-15T21:12:24.960132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:24.960132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17103518,"num_objects":2089,"num_object_clones":0,"num_object_copies":8356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2089,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10343,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-169}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10500","reported_seq":10575,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.293760+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:27.293760+0000","last_peered":"2024-09-15T21:12:27.293760+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:27.293760+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16020748,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10498,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-378}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10679","reported_seq":10740,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.925652+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:28.925652+0000","last_peered":"2024-09-15T21:12:28.925652+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:28.925652+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16217356,"num_objects":1980,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10677,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-331}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10429","reported_seq":10480,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.737552+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:28.737552+0000","last_peered":"2024-09-15T21:12:28.737552+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:28.737552+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15949824,"num_objects":1947,"num_object_clones":0,"num_object_copies":7788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10429,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-369}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10704","reported_seq":10780,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.428659+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:27.428659+0000","last_peered":"2024-09-15T21:12:27.428659+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:27.428659+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16938252,"num_objects":2068,"num_object_clones":0,"num_object_copies":8272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10750","reported_seq":10825,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.337570+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:27.337570+0000","last_peered":"2024-09-15T21:12:27.337570+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:27.337570+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16471308,"num_objects":2011,"num_object_clones":0,"num_object_copies":8044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10748,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-463}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10478","reported_seq":10544,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.925500+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:28.925500+0000","last_peered":"2024-09-15T21:12:28.925500+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:28.925500+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16165400,"num_objects":1974,"num_object_clones":0,"num_object_copies":7896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10475,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10496","reported_seq":10583,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:24.914664+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:24.914664+0000","last_peered":"2024-09-15T21:12:24.914664+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:24.914664+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16921868,"num_objects":2066,"num_object_clones":0,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-269}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10394","reported_seq":15062,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:28.693090+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:28.693090+0000","last_peered":"2024-09-15T21:12:28.693090+0000","last_clean":"2024-09-15T21:12:28.693090+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:28.693090+0000","last_undegraded":"2024-09-15T21:12:28.693090+0000","last_fullsized":"2024-09-15T21:12:28.693090+0000","mapping_epoch":32,"log_start":"27'300","ondisk_log_start":"27'300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16105472,"num_objects":1966,"num_object_clones":0,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4505,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10419","reported_seq":10465,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.983429+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:28.983429+0000","last_peered":"2024-09-15T21:12:28.983429+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:28.983429+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16448134,"num_objects":2008,"num_object_clones":0,"num_object_copies":8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10418,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10432","reported_seq":12671,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.265627+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:25.265627+0000","last_peered":"2024-09-15T21:12:25.265627+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:25.265627+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16947846,"num_objects":2069,"num_object_clones":0,"num_object_copies":8276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1211,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10431,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10492","reported_seq":10555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:28.838402+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:28.838402+0000","last_peered":"2024-09-15T21:12:28.838402+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:28.838402+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15889676,"num_objects":1940,"num_object_clones":0,"num_object_copies":7760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10490,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-339}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10516","reported_seq":10604,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:25.153286+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:25.153286+0000","last_peered":"2024-09-15T21:12:25.153286+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:25.153286+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17732876,"num_objects":2165,"num_object_clones":0,"num_object_copies":8660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10514,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-116}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10556","reported_seq":10631,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:27.419282+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:27.419282+0000","last_peered":"2024-09-15T21:12:27.419282+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:27.419282+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16323852,"num_objects":1993,"num_object_clones":0,"num_object_copies":7972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-322}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":263725602,"num_objects":32198,"num_object_clones":0,"num_object_copies":128792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152880,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1098,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":528588800,"data_stored":528588800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160825,"ondisk_log_size":160825,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728224,"kb_used_data":128716,"kb_used_omap":4,"kb_used_meta":599483,"kb_avail":93643616,"statfs":{"total":96636764160,"available":95891062784,"internally_reserved":0,"allocated":131805184,"data_stored":131632082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":613871312},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,2,4,1,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":730132,"kb_used_data":131140,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":93641708,"statfs":{"total":96636764160,"available":95889108992,"internally_reserved":0,"allocated":134287360,"data_stored":134114258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051686,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":602980,"kb_used_data":129876,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93768860,"statfs":{"total":96636764160,"available":96019312640,"internally_reserved":0,"allocated":132993024,"data_stored":132819922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4,2,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247924,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":738920,"kb_used_data":129752,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93632920,"statfs":{"total":96636764160,"available":95880110080,"internally_reserved":0,"allocated":132866048,"data_stored":132692946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":623767246},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,0,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":131899392,"data_stored":131899392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132186112,"data_stored":132186112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":133808128,"data_stored":133808128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":130695168,"data_stored":130695168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:33.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:34.520 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:34.520 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:34.546 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-15T21:12:34.146776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255902242,"num_objects":31243,"num_object_clones":0,"num_object_copies":124972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153835,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804860,"kb_used_data":503212,"kb_used_omap":14,"kb_used_meta":2301553,"kb_avail":374682500,"statfs":{"total":386547056640,"available":383674880000,"internally_reserved":0,"allocated":515289088,"data_stored":514596680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2356790834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,4,8,11,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21716992,"num_objects":-2651,"num_object_clones":0,"num_object_copies":-10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004485"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.879926+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:33.879926+0000","last_peered":"2024-09-15T21:12:33.879926+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:33.879926+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15425536,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10594","reported_seq":15487,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:32.208941+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:32.208941+0000","last_peered":"2024-09-15T21:12:32.208941+0000","last_clean":"2024-09-15T21:12:32.208941+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:32.208941+0000","last_undegraded":"2024-09-15T21:12:32.208941+0000","last_fullsized":"2024-09-15T21:12:32.208941+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16023552,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10415","reported_seq":10501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.161653+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:30.161653+0000","last_peered":"2024-09-15T21:12:30.161653+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:30.161653+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16521886,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10414,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10568","reported_seq":10641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.400396+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:32.400396+0000","last_peered":"2024-09-15T21:12:32.400396+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:32.400396+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463692,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10732","reported_seq":10794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.696614+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:33.696614+0000","last_peered":"2024-09-15T21:12:33.696614+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:33.696614+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15783180,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10489","reported_seq":10540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.446485+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:33.446485+0000","last_peered":"2024-09-15T21:12:33.446485+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:33.446485+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15458304,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10757","reported_seq":10833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.324749+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:32.324749+0000","last_peered":"2024-09-15T21:12:32.324749+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:32.324749+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16504076,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10813","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.342358+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:32.342358+0000","last_peered":"2024-09-15T21:12:32.342358+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:32.342358+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955212,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10529","reported_seq":10595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.729264+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:33.729264+0000","last_peered":"2024-09-15T21:12:33.729264+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:33.729264+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15747608,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10537","reported_seq":10624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.099833+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:30.099833+0000","last_peered":"2024-09-15T21:12:30.099833+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:30.099833+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16585996,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10453","reported_seq":15122,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:33.945441+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:33.945441+0000","last_peered":"2024-09-15T21:12:33.945441+0000","last_clean":"2024-09-15T21:12:33.945441+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:33.945441+0000","last_undegraded":"2024-09-15T21:12:33.945441+0000","last_fullsized":"2024-09-15T21:12:33.945441+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15622144,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10470","reported_seq":10516,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.880699+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:33.880699+0000","last_peered":"2024-09-15T21:12:33.880699+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:33.880699+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16030342,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10495","reported_seq":12860,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.284832+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:30.284832+0000","last_peered":"2024-09-15T21:12:30.284832+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:30.284832+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16431750,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10563","reported_seq":10624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:34.038973+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:34.038973+0000","last_peered":"2024-09-15T21:12:34.038973+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:34.038973+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15308044,"num_objects":1869,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10581","reported_seq":10670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.255317+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:30.255317+0000","last_peered":"2024-09-15T21:12:30.255317+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:30.255317+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17200396,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10615","reported_seq":10690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.462936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:32.462936+0000","last_peered":"2024-09-15T21:12:32.462936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:32.462936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15840524,"num_objects":1934,"num_object_clones":0,"num_object_copies":7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":255902242,"num_objects":31243,"num_object_clones":0,"num_object_copies":124972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153835,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512499712,"data_stored":512499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729120,"kb_used_data":124048,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93642720,"statfs":{"total":96636764160,"available":95890145280,"internally_reserved":0,"allocated":127025152,"data_stored":126852050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731816,"kb_used_data":127256,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93640024,"statfs":{"total":96636764160,"available":95887384576,"internally_reserved":0,"allocated":130310144,"data_stored":130137042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586668,"kb_used_data":126040,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93785172,"statfs":{"total":96636764160,"available":96036016128,"internally_reserved":0,"allocated":129064960,"data_stored":128891858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757256,"kb_used_data":125868,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93614584,"statfs":{"total":96636764160,"available":95861334016,"internally_reserved":0,"allocated":128888832,"data_stored":128715730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127942656,"data_stored":127942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129716224,"data_stored":129716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126676992,"data_stored":126676992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:34.548 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:35.124 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:35.124 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:35.151 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-15T21:12:34.146776+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":255902242,"num_objects":31243,"num_object_clones":0,"num_object_copies":124972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153835,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2804860,"kb_used_data":503212,"kb_used_omap":14,"kb_used_meta":2301553,"kb_avail":374682500,"statfs":{"total":386547056640,"available":383674880000,"internally_reserved":0,"allocated":515289088,"data_stored":514596680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2356790834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,4,8,11,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21716992,"num_objects":-2651,"num_object_clones":0,"num_object_copies":-10604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-120,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2651,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004485"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.879926+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:33.879926+0000","last_peered":"2024-09-15T21:12:33.879926+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:33.879926+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15425536,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10594","reported_seq":15487,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:32.208941+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:32.208941+0000","last_peered":"2024-09-15T21:12:32.208941+0000","last_clean":"2024-09-15T21:12:32.208941+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:32.208941+0000","last_undegraded":"2024-09-15T21:12:32.208941+0000","last_fullsized":"2024-09-15T21:12:32.208941+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16023552,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10415","reported_seq":10501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.161653+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:30.161653+0000","last_peered":"2024-09-15T21:12:30.161653+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:30.161653+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16521886,"num_objects":2018,"num_object_clones":0,"num_object_copies":8072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10414,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-240}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10568","reported_seq":10641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.400396+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:32.400396+0000","last_peered":"2024-09-15T21:12:32.400396+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:32.400396+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463692,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10732","reported_seq":10794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.696614+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:33.696614+0000","last_peered":"2024-09-15T21:12:33.696614+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:33.696614+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15783180,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10489","reported_seq":10540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.446485+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:33.446485+0000","last_peered":"2024-09-15T21:12:33.446485+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:33.446485+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15458304,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10757","reported_seq":10833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.324749+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:32.324749+0000","last_peered":"2024-09-15T21:12:32.324749+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:32.324749+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16504076,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10813","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.342358+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:32.342358+0000","last_peered":"2024-09-15T21:12:32.342358+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:32.342358+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955212,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10529","reported_seq":10595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.729264+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:33.729264+0000","last_peered":"2024-09-15T21:12:33.729264+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:33.729264+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15747608,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10537","reported_seq":10624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.099833+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:30.099833+0000","last_peered":"2024-09-15T21:12:30.099833+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:30.099833+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16585996,"num_objects":2025,"num_object_clones":0,"num_object_copies":8100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10535,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-310}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10453","reported_seq":15122,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:33.945441+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:33.945441+0000","last_peered":"2024-09-15T21:12:33.945441+0000","last_clean":"2024-09-15T21:12:33.945441+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:33.945441+0000","last_undegraded":"2024-09-15T21:12:33.945441+0000","last_fullsized":"2024-09-15T21:12:33.945441+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15622144,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10470","reported_seq":10516,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.880699+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:33.880699+0000","last_peered":"2024-09-15T21:12:33.880699+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:33.880699+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16030342,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10495","reported_seq":12860,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.284832+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:30.284832+0000","last_peered":"2024-09-15T21:12:30.284832+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:30.284832+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16431750,"num_objects":2006,"num_object_clones":0,"num_object_copies":8024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10494,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1140,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":858}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10563","reported_seq":10624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:34.038973+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:34.038973+0000","last_peered":"2024-09-15T21:12:34.038973+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:34.038973+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15308044,"num_objects":1869,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10581","reported_seq":10670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:30.255317+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:30.255317+0000","last_peered":"2024-09-15T21:12:30.255317+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:30.255317+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17200396,"num_objects":2100,"num_object_clones":0,"num_object_copies":8400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10579,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-181}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10615","reported_seq":10690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.462936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:32.462936+0000","last_peered":"2024-09-15T21:12:32.462936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:32.462936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15840524,"num_objects":1934,"num_object_clones":0,"num_object_copies":7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":255902242,"num_objects":31243,"num_object_clones":0,"num_object_copies":124972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31243,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":153835,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512499712,"data_stored":512499712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160880,"ondisk_log_size":160880,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729120,"kb_used_data":124048,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93642720,"statfs":{"total":96636764160,"available":95890145280,"internally_reserved":0,"allocated":127025152,"data_stored":126852050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731816,"kb_used_data":127256,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93640024,"statfs":{"total":96636764160,"available":95887384576,"internally_reserved":0,"allocated":130310144,"data_stored":130137042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586668,"kb_used_data":126040,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93785172,"statfs":{"total":96636764160,"available":96036016128,"internally_reserved":0,"allocated":129064960,"data_stored":128891858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757256,"kb_used_data":125868,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93614584,"statfs":{"total":96636764160,"available":95861334016,"internally_reserved":0,"allocated":128888832,"data_stored":128715730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127942656,"data_stored":127942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":129716224,"data_stored":129716224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126676992,"data_stored":126676992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:35.152 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:35.152 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10469", 'reported_seq': 10521, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:33.879926+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:33.879926+0000', 'last_peered': '2024-09-15T21:12:33.879926+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:33.879926+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15425536, 'num_objects': 1883, 'num_object_clones': 0, 'num_object_copies': 7532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1883, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10469, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -393}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10415", 'reported_seq': 10501, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:30.161653+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:30.161653+0000', 'last_peered': '2024-09-15T21:12:30.161653+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:30.161653+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16521886, 'num_objects': 2018, 'num_object_clones': 0, 'num_object_copies': 8072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2018, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10414, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -240}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10568", 'reported_seq': 10641, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:32.400396+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:32.400396+0000', 'last_peered': '2024-09-15T21:12:32.400396+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:32.400396+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15463692, 'num_objects': 1888, 'num_object_clones': 0, 'num_object_copies': 7552, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1888, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10566, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -446}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10732", 'reported_seq': 10794, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:33.696614+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:33.696614+0000', 'last_peered': '2024-09-15T21:12:33.696614+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:33.696614+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15783180, 'num_objects': 1927, 'num_object_clones': 0, 'num_object_copies': 7708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10730, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -384}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10489", 'reported_seq': 10540, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:33.446485+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:33.446485+0000', 'last_peered': '2024-09-15T21:12:33.446485+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:33.446485+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15458304, 'num_objects': 1887, 'num_object_clones': 0, 'num_object_copies': 7548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10489, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -429}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.153 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10757", 'reported_seq': 10833, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:32.324749+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:32.324749+0000', 'last_peered': '2024-09-15T21:12:32.324749+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:32.324749+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16504076, 'num_objects': 2015, 'num_object_clones': 0, 'num_object_copies': 8060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2015, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10755, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -351}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10813", 'reported_seq': 10888, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:32.342358+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:32.342358+0000', 'last_peered': '2024-09-15T21:12:32.342358+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:32.342358+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15955212, 'num_objects': 1948, 'num_object_clones': 0, 'num_object_copies': 7792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1948, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10811, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -526}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10529", 'reported_seq': 10595, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:33.729264+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:33.729264+0000', 'last_peered': '2024-09-15T21:12:33.729264+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:33.729264+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15747608, 'num_objects': 1923, 'num_object_clones': 0, 'num_object_copies': 7692, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1923, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10526, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -409}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10537", 'reported_seq': 10624, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:30.099833+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:30.099833+0000', 'last_peered': '2024-09-15T21:12:30.099833+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:30.099833+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16585996, 'num_objects': 2025, 'num_object_clones': 0, 'num_object_copies': 8100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2025, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10535, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -310}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10470", 'reported_seq': 10516, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:33.880699+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:33.880699+0000', 'last_peered': '2024-09-15T21:12:33.880699+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:33.880699+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16030342, 'num_objects': 1957, 'num_object_clones': 0, 'num_object_copies': 7828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1957, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10469, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -299}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.154 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10495", 'reported_seq': 12860, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:30.284832+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:30.284832+0000', 'last_peered': '2024-09-15T21:12:30.284832+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:30.284832+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16431750, 'num_objects': 2006, 'num_object_clones': 0, 'num_object_copies': 8024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1148, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2006, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10494, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1140, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10563", 'reported_seq': 10624, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:34.038973+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:34.038973+0000', 'last_peered': '2024-09-15T21:12:34.038973+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:34.038973+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15308044, 'num_objects': 1869, 'num_object_clones': 0, 'num_object_copies': 7476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1869, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10561, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -410}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10581", 'reported_seq': 10670, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:30.255317+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:30.255317+0000', 'last_peered': '2024-09-15T21:12:30.255317+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:30.255317+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 17200396, 'num_objects': 2100, 'num_object_clones': 0, 'num_object_copies': 8400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2100, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10579, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -181}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:35.155 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10615", 'reported_seq': 10690, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:32.462936+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:32.462936+0000', 'last_peered': '2024-09-15T21:12:32.462936+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:32.462936+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15840524, 'num_objects': 1934, 'num_object_clones': 0, 'num_object_copies': 7736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1934, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10613, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -381}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:35.156 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:35.925 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:35.925 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31302/124972 objects degraded (25.047%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":621,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":31243,"data_bytes":255902242,"bytes_used":2872176640,"bytes_avail":383674880000,"bytes_total":386547056640,"degraded_objects":31302,"degraded_total":124972,"degraded_ratio":0.25047210575168838,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":120,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":220},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:35.946 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:35.947 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:36.556 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:36.557 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:36.590 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-15T21:12:36.147530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":253764130,"num_objects":30982,"num_object_clones":0,"num_object_copies":123928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154096,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2805088,"kb_used_data":499276,"kb_used_omap":14,"kb_used_meta":2305713,"kb_avail":374682272,"statfs":{"total":386547056640,"available":383674646528,"internally_reserved":0,"allocated":511258624,"data_stored":510566216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2361050674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,5,3,5,12,8,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17965056,"num_objects":-2193,"num_object_clones":0,"num_object_copies":-8772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2193,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004603"},"pg_stats":[{"pgid":"1.f","version":"35'10469","reported_seq":10521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.879926+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:33.879926+0000","last_peered":"2024-09-15T21:12:33.879926+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:33.879926+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15425536,"num_objects":1883,"num_object_clones":0,"num_object_copies":7532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-393}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10594","reported_seq":15487,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:32.208941+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:32.208941+0000","last_peered":"2024-09-15T21:12:32.208941+0000","last_clean":"2024-09-15T21:12:32.208941+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:32.208941+0000","last_undegraded":"2024-09-15T21:12:32.208941+0000","last_fullsized":"2024-09-15T21:12:32.208941+0000","mapping_epoch":34,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16023552,"num_objects":1956,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10476","reported_seq":10562,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.178299+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:35.178299+0000","last_peered":"2024-09-15T21:12:35.178299+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:35.178299+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16022174,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10475,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10568","reported_seq":10641,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.400396+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:32.400396+0000","last_peered":"2024-09-15T21:12:32.400396+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:32.400396+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15463692,"num_objects":1888,"num_object_clones":0,"num_object_copies":7552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-446}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10732","reported_seq":10794,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.696614+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:33.696614+0000","last_peered":"2024-09-15T21:12:33.696614+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:33.696614+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15783180,"num_objects":1927,"num_object_clones":0,"num_object_copies":7708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-384}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10489","reported_seq":10540,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.446485+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:33.446485+0000","last_peered":"2024-09-15T21:12:33.446485+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:33.446485+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15458304,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10489,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-429}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10757","reported_seq":10833,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.324749+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:32.324749+0000","last_peered":"2024-09-15T21:12:32.324749+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:32.324749+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16504076,"num_objects":2015,"num_object_clones":0,"num_object_copies":8060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10755,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-351}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10813","reported_seq":10888,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.342358+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:32.342358+0000","last_peered":"2024-09-15T21:12:32.342358+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:32.342358+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15955212,"num_objects":1948,"num_object_clones":0,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-526}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10529","reported_seq":10595,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.729264+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:33.729264+0000","last_peered":"2024-09-15T21:12:33.729264+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:33.729264+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15747608,"num_objects":1923,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10526,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-409}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10608","reported_seq":10695,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.253091+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:35.253091+0000","last_peered":"2024-09-15T21:12:35.253091+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:35.253091+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10453","reported_seq":15122,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:33.945441+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:33.945441+0000","last_peered":"2024-09-15T21:12:33.945441+0000","last_clean":"2024-09-15T21:12:33.945441+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:33.945441+0000","last_undegraded":"2024-09-15T21:12:33.945441+0000","last_fullsized":"2024-09-15T21:12:33.945441+0000","mapping_epoch":32,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15622144,"num_objects":1907,"num_object_clones":0,"num_object_copies":7628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4564,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10470","reported_seq":10516,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:33.880699+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:33.880699+0000","last_peered":"2024-09-15T21:12:33.880699+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:33.880699+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16030342,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10469,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-299}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10559","reported_seq":13044,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.260487+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:35.260487+0000","last_peered":"2024-09-15T21:12:35.260487+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:35.260487+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15907462,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1201,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10563","reported_seq":10624,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:34.038973+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:34.038973+0000","last_peered":"2024-09-15T21:12:34.038973+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:34.038973+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15308044,"num_objects":1869,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-410}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10646","reported_seq":10735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.205946+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:35.205946+0000","last_peered":"2024-09-15T21:12:35.205946+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:35.205946+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16667916,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10615","reported_seq":10690,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:32.462936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:32.462936+0000","last_peered":"2024-09-15T21:12:32.462936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:32.462936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15840524,"num_objects":1934,"num_object_clones":0,"num_object_copies":7736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-381}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":253764130,"num_objects":30982,"num_object_clones":0,"num_object_copies":123928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154096,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":508448768,"data_stored":508448768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160841,"ondisk_log_size":160841,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729120,"kb_used_data":124048,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93642720,"statfs":{"total":96636764160,"available":95890145280,"internally_reserved":0,"allocated":127025152,"data_stored":126852050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732044,"kb_used_data":123320,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93639796,"statfs":{"total":96636764160,"available":95887151104,"internally_reserved":0,"allocated":126279680,"data_stored":126106578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051687,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586668,"kb_used_data":126040,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93785172,"statfs":{"total":96636764160,"available":96036016128,"internally_reserved":0,"allocated":129064960,"data_stored":128891858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757256,"kb_used_data":125868,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93614584,"statfs":{"total":96636764160,"available":95861334016,"internally_reserved":0,"allocated":128888832,"data_stored":128715730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,1,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":127942656,"data_stored":127942656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":128163840,"data_stored":128163840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125665280,"data_stored":125665280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":126676992,"data_stored":126676992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:39.592 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:40.316 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:40.316 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:40.345 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-15T21:12:40.148803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247816738,"num_objects":30256,"num_object_clones":0,"num_object_copies":121024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154822,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806072,"kb_used_data":487796,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374681288,"statfs":{"total":386547056640,"available":383673638912,"internally_reserved":0,"allocated":499503104,"data_stored":498810696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2373830194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,3,6,7,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19546112,"num_objects":-2386,"num_object_clones":0,"num_object_copies":-9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004291"},"pg_stats":[{"pgid":"1.f","version":"35'10534","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.123293+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:39.123293+0000","last_peered":"2024-09-15T21:12:39.123293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:39.123293+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893056,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10659","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:37.495057+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:37.495057+0000","last_peered":"2024-09-15T21:12:37.495057+0000","last_clean":"2024-09-15T21:12:37.495057+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:37.495057+0000","last_undegraded":"2024-09-15T21:12:37.495057+0000","last_fullsized":"2024-09-15T21:12:37.495057+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10476","reported_seq":10562,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.178299+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:35.178299+0000","last_peered":"2024-09-15T21:12:35.178299+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:35.178299+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16022174,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10475,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10623","reported_seq":10698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.347211+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:37.347211+0000","last_peered":"2024-09-15T21:12:37.347211+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:37.347211+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15013132,"num_objects":1833,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10621,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10780","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.035143+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:39.035143+0000","last_peered":"2024-09-15T21:12:39.035143+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:39.035143+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389964,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10559","reported_seq":10608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.190023+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:39.190023+0000","last_peered":"2024-09-15T21:12:39.190023+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:39.190023+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14884864,"num_objects":1817,"num_object_clones":0,"num_object_copies":7268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10831","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.446157+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:37.446157+0000","last_peered":"2024-09-15T21:12:37.446157+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:37.446157+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15897868,"num_objects":1941,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10876","reported_seq":10951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.334919+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:37.334919+0000","last_peered":"2024-09-15T21:12:37.334919+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:37.334919+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15439116,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10587","reported_seq":10653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.045018+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:39.045018+0000","last_peered":"2024-09-15T21:12:39.045018+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:39.045018+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15272472,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10608","reported_seq":10695,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.253091+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:35.253091+0000","last_peered":"2024-09-15T21:12:35.253091+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:35.253091+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10519","reported_seq":15188,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:39.035011+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:39.035011+0000","last_peered":"2024-09-15T21:12:39.035011+0000","last_clean":"2024-09-15T21:12:39.035011+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:39.035011+0000","last_undegraded":"2024-09-15T21:12:39.035011+0000","last_fullsized":"2024-09-15T21:12:39.035011+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15081472,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10524","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.121943+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:39.121943+0000","last_peered":"2024-09-15T21:12:39.121943+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:39.121943+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15587974,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10559","reported_seq":13044,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.260487+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:35.260487+0000","last_peered":"2024-09-15T21:12:35.260487+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:35.260487+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15907462,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1201,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10624","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.039359+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:39.039359+0000","last_peered":"2024-09-15T21:12:39.039359+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:39.039359+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808332,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10646","reported_seq":10735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.205946+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:35.205946+0000","last_peered":"2024-09-15T21:12:35.205946+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:35.205946+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16667916,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10662","reported_seq":10737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.540153+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:37.540153+0000","last_peered":"2024-09-15T21:12:37.540153+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:37.540153+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15455500,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":247816738,"num_objects":30256,"num_object_clones":0,"num_object_copies":121024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154822,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":496226304,"data_stored":496226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729444,"kb_used_data":120212,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93642396,"statfs":{"total":96636764160,"available":95889813504,"internally_reserved":0,"allocated":123097088,"data_stored":122923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732044,"kb_used_data":123320,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93639796,"statfs":{"total":96636764160,"available":95887151104,"internally_reserved":0,"allocated":126279680,"data_stored":126106578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587048,"kb_used_data":122264,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93784792,"statfs":{"total":96636764160,"available":96035627008,"internally_reserved":0,"allocated":125198336,"data_stored":125025234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":122000,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":124928000,"data_stored":124754898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123944960,"data_stored":123944960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124047360,"data_stored":124047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125665280,"data_stored":125665280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122568704,"data_stored":122568704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:40.347 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:40.936 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:40.936 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:40.960 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-15T21:12:40.148803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247816738,"num_objects":30256,"num_object_clones":0,"num_object_copies":121024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154822,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806072,"kb_used_data":487796,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374681288,"statfs":{"total":386547056640,"available":383673638912,"internally_reserved":0,"allocated":499503104,"data_stored":498810696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2373830194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,3,6,7,15,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19546112,"num_objects":-2386,"num_object_clones":0,"num_object_copies":-9544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2386,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004291"},"pg_stats":[{"pgid":"1.f","version":"35'10534","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.123293+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:39.123293+0000","last_peered":"2024-09-15T21:12:39.123293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:39.123293+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893056,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10659","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:37.495057+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:37.495057+0000","last_peered":"2024-09-15T21:12:37.495057+0000","last_clean":"2024-09-15T21:12:37.495057+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:37.495057+0000","last_undegraded":"2024-09-15T21:12:37.495057+0000","last_fullsized":"2024-09-15T21:12:37.495057+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10476","reported_seq":10562,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.178299+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:35.178299+0000","last_peered":"2024-09-15T21:12:35.178299+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:35.178299+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'400","ondisk_log_start":"27'400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16022174,"num_objects":1957,"num_object_clones":0,"num_object_copies":7828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1957,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10475,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-301}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10623","reported_seq":10698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.347211+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:37.347211+0000","last_peered":"2024-09-15T21:12:37.347211+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:37.347211+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15013132,"num_objects":1833,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10621,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10780","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.035143+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:39.035143+0000","last_peered":"2024-09-15T21:12:39.035143+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:39.035143+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389964,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10559","reported_seq":10608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.190023+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:39.190023+0000","last_peered":"2024-09-15T21:12:39.190023+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:39.190023+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14884864,"num_objects":1817,"num_object_clones":0,"num_object_copies":7268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10831","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.446157+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:37.446157+0000","last_peered":"2024-09-15T21:12:37.446157+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:37.446157+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15897868,"num_objects":1941,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10876","reported_seq":10951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.334919+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:37.334919+0000","last_peered":"2024-09-15T21:12:37.334919+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:37.334919+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15439116,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10587","reported_seq":10653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.045018+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:39.045018+0000","last_peered":"2024-09-15T21:12:39.045018+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:39.045018+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15272472,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10608","reported_seq":10695,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.253091+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:35.253091+0000","last_peered":"2024-09-15T21:12:35.253091+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:35.253091+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16004364,"num_objects":1954,"num_object_clones":0,"num_object_copies":7816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-381}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10519","reported_seq":15188,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:39.035011+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:39.035011+0000","last_peered":"2024-09-15T21:12:39.035011+0000","last_clean":"2024-09-15T21:12:39.035011+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:39.035011+0000","last_undegraded":"2024-09-15T21:12:39.035011+0000","last_fullsized":"2024-09-15T21:12:39.035011+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15081472,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10524","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.121943+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:39.121943+0000","last_peered":"2024-09-15T21:12:39.121943+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:39.121943+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15587974,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10559","reported_seq":13044,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.260487+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:35.260487+0000","last_peered":"2024-09-15T21:12:35.260487+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:35.260487+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15907462,"num_objects":1942,"num_object_clones":0,"num_object_copies":7768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1087,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1201,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":855}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10624","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.039359+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:39.039359+0000","last_peered":"2024-09-15T21:12:39.039359+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:39.039359+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808332,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10646","reported_seq":10735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:35.205946+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:35.205946+0000","last_peered":"2024-09-15T21:12:35.205946+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:35.205946+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16667916,"num_objects":2035,"num_object_clones":0,"num_object_copies":8140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-246}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10662","reported_seq":10737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.540153+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:37.540153+0000","last_peered":"2024-09-15T21:12:37.540153+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:37.540153+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15455500,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":247816738,"num_objects":30256,"num_object_clones":0,"num_object_copies":121024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":154822,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":496226304,"data_stored":496226304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160767,"ondisk_log_size":160767,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729444,"kb_used_data":120212,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93642396,"statfs":{"total":96636764160,"available":95889813504,"internally_reserved":0,"allocated":123097088,"data_stored":122923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732044,"kb_used_data":123320,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93639796,"statfs":{"total":96636764160,"available":95887151104,"internally_reserved":0,"allocated":126279680,"data_stored":126106578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587048,"kb_used_data":122264,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93784792,"statfs":{"total":96636764160,"available":96035627008,"internally_reserved":0,"allocated":125198336,"data_stored":125025234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":122000,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":124928000,"data_stored":124754898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123944960,"data_stored":123944960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124047360,"data_stored":124047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125665280,"data_stored":125665280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122568704,"data_stored":122568704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10534", 'reported_seq': 10586, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.123293+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:39.123293+0000', 'last_peered': '2024-09-15T21:12:39.123293+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:39.123293+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14893056, 'num_objects': 1818, 'num_object_clones': 0, 'num_object_copies': 7272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10534, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -458}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10476", 'reported_seq': 10562, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:35.178299+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:35.178299+0000', 'last_peered': '2024-09-15T21:12:35.178299+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:35.178299+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'400", 'ondisk_log_start': "27'400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16022174, 'num_objects': 1957, 'num_object_clones': 0, 'num_object_copies': 7828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1957, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10475, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -301}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:40.961 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10623", 'reported_seq': 10698, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:37.347211+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:37.347211+0000', 'last_peered': '2024-09-15T21:12:37.347211+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:37.347211+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15013132, 'num_objects': 1833, 'num_object_clones': 0, 'num_object_copies': 7332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10621, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -501}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10780", 'reported_seq': 10842, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.035143+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:39.035143+0000', 'last_peered': '2024-09-15T21:12:39.035143+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:39.035143+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15389964, 'num_objects': 1879, 'num_object_clones': 0, 'num_object_copies': 7516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1879, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10778, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -432}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10559", 'reported_seq': 10608, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.190023+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:39.190023+0000', 'last_peered': '2024-09-15T21:12:39.190023+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:39.190023+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14884864, 'num_objects': 1817, 'num_object_clones': 0, 'num_object_copies': 7268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1817, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10559, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10831", 'reported_seq': 10907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:37.446157+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:37.446157+0000', 'last_peered': '2024-09-15T21:12:37.446157+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:37.446157+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15897868, 'num_objects': 1941, 'num_object_clones': 0, 'num_object_copies': 7764, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1941, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10829, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -425}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10876", 'reported_seq': 10951, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:37.334919+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:37.334919+0000', 'last_peered': '2024-09-15T21:12:37.334919+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:37.334919+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15439116, 'num_objects': 1885, 'num_object_clones': 0, 'num_object_copies': 7540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1885, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10874, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -589}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:40.962 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10587", 'reported_seq': 10653, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.045018+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:39.045018+0000', 'last_peered': '2024-09-15T21:12:39.045018+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:39.045018+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15272472, 'num_objects': 1865, 'num_object_clones': 0, 'num_object_copies': 7460, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1865, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10584, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -467}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10608", 'reported_seq': 10695, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:35.253091+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:35.253091+0000', 'last_peered': '2024-09-15T21:12:35.253091+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:35.253091+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16004364, 'num_objects': 1954, 'num_object_clones': 0, 'num_object_copies': 7816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1954, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10606, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -381}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10524", 'reported_seq': 10570, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.121943+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:39.121943+0000', 'last_peered': '2024-09-15T21:12:39.121943+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:39.121943+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15587974, 'num_objects': 1903, 'num_object_clones': 0, 'num_object_copies': 7612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1903, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10523, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -353}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:40.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10559", 'reported_seq': 13044, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:35.260487+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:35.260487+0000', 'last_peered': '2024-09-15T21:12:35.260487+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:35.260487+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15907462, 'num_objects': 1942, 'num_object_clones': 0, 'num_object_copies': 7768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1087, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10558, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1201, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 855}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10624", 'reported_seq': 10687, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:39.039359+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:39.039359+0000', 'last_peered': '2024-09-15T21:12:39.039359+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:39.039359+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14808332, 'num_objects': 1808, 'num_object_clones': 0, 'num_object_copies': 7232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1808, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10622, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -471}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10646", 'reported_seq': 10735, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:35.205946+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:35.205946+0000', 'last_peered': '2024-09-15T21:12:35.205946+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:35.205946+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16667916, 'num_objects': 2035, 'num_object_clones': 0, 'num_object_copies': 8140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2035, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10644, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -246}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:40.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10662", 'reported_seq': 10737, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:37.540153+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:37.540153+0000', 'last_peered': '2024-09-15T21:12:37.540153+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:37.540153+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15455500, 'num_objects': 1887, 'num_object_clones': 0, 'num_object_copies': 7548, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1887, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10660, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -428}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:40.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:41.748 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:41.748 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31241/121024 objects degraded (25.814%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":627,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":30256,"data_bytes":247816738,"bytes_used":2873417728,"bytes_avail":383673638912,"bytes_total":386547056640,"degraded_objects":31241,"degraded_total":121024,"degraded_ratio":0.25813888154415654,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":198},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:41.769 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:41.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:42.360 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:42.361 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:42.382 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-09-15T21:12:42.149619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":245539362,"num_objects":29978,"num_object_clones":0,"num_object_copies":119912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155100,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2806340,"kb_used_data":483908,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374681020,"statfs":{"total":386547056640,"available":383673364480,"internally_reserved":0,"allocated":495521792,"data_stored":494829384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2378090034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,2,8,5,7,18,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18186240,"num_objects":-2220,"num_object_clones":0,"num_object_copies":-8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004418"},"pg_stats":[{"pgid":"1.f","version":"35'10534","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.123293+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:39.123293+0000","last_peered":"2024-09-15T21:12:39.123293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:39.123293+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893056,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10659","reported_seq":15552,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:37.495057+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:37.495057+0000","last_peered":"2024-09-15T21:12:37.495057+0000","last_clean":"2024-09-15T21:12:37.495057+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:37.495057+0000","last_undegraded":"2024-09-15T21:12:37.495057+0000","last_fullsized":"2024-09-15T21:12:37.495057+0000","mapping_epoch":34,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15491072,"num_objects":1891,"num_object_clones":0,"num_object_copies":7564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1325,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10532","reported_seq":10618,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.235292+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:40.235292+0000","last_peered":"2024-09-15T21:12:40.235292+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:40.235292+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563422,"num_objects":1901,"num_object_clones":0,"num_object_copies":7604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10531,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10623","reported_seq":10698,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.347211+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:37.347211+0000","last_peered":"2024-09-15T21:12:37.347211+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:37.347211+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15013132,"num_objects":1833,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10621,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-501}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10780","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.035143+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:39.035143+0000","last_peered":"2024-09-15T21:12:39.035143+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:39.035143+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389964,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10559","reported_seq":10608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.190023+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:39.190023+0000","last_peered":"2024-09-15T21:12:39.190023+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:39.190023+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14884864,"num_objects":1817,"num_object_clones":0,"num_object_copies":7268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10831","reported_seq":10907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.446157+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:37.446157+0000","last_peered":"2024-09-15T21:12:37.446157+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:37.446157+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15897868,"num_objects":1941,"num_object_clones":0,"num_object_copies":7764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10829,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-425}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10876","reported_seq":10951,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.334919+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:37.334919+0000","last_peered":"2024-09-15T21:12:37.334919+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:37.334919+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15439116,"num_objects":1885,"num_object_clones":0,"num_object_copies":7540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10874,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-589}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10587","reported_seq":10653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.045018+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:39.045018+0000","last_peered":"2024-09-15T21:12:39.045018+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:39.045018+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15272472,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10680","reported_seq":10767,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.346778+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:40.346778+0000","last_peered":"2024-09-15T21:12:40.346778+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:40.346778+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15414540,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10519","reported_seq":15188,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:39.035011+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:39.035011+0000","last_peered":"2024-09-15T21:12:39.035011+0000","last_clean":"2024-09-15T21:12:39.035011+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:39.035011+0000","last_undegraded":"2024-09-15T21:12:39.035011+0000","last_fullsized":"2024-09-15T21:12:39.035011+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15081472,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10524","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.121943+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:39.121943+0000","last_peered":"2024-09-15T21:12:39.121943+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:39.121943+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15587974,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10635","reported_seq":13226,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.389209+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:40.389209+0000","last_peered":"2024-09-15T21:12:40.389209+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:40.389209+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15284870,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10634,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10624","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.039359+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:39.039359+0000","last_peered":"2024-09-15T21:12:39.039359+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:39.039359+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808332,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10720","reported_seq":10806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.113713+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:40.113713+0000","last_peered":"2024-09-15T21:12:40.113713+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:40.113713+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16061708,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-320}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10662","reported_seq":10737,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:37.540153+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:37.540153+0000","last_peered":"2024-09-15T21:12:37.540153+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:37.540153+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15455500,"num_objects":1887,"num_object_clones":0,"num_object_copies":7548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-428}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":245539362,"num_objects":29978,"num_object_clones":0,"num_object_copies":119912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155100,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":492158976,"data_stored":492158976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160745,"ondisk_log_size":160745,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729444,"kb_used_data":120212,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93642396,"statfs":{"total":96636764160,"available":95889813504,"internally_reserved":0,"allocated":123097088,"data_stored":122923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732312,"kb_used_data":119432,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93639528,"statfs":{"total":96636764160,"available":95886876672,"internally_reserved":0,"allocated":122298368,"data_stored":122125266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051688,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587048,"kb_used_data":122264,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93784792,"statfs":{"total":96636764160,"available":96035627008,"internally_reserved":0,"allocated":125198336,"data_stored":125025234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,4,6,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247926,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":757536,"kb_used_data":122000,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93614304,"statfs":{"total":96636764160,"available":95861047296,"internally_reserved":0,"allocated":124928000,"data_stored":124754898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,3,1,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":123944960,"data_stored":123944960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":124047360,"data_stored":124047360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121597952,"data_stored":121597952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122568704,"data_stored":122568704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:45.384 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:46.005 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:46.006 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:46.040 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-15T21:12:44.150310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":243065378,"num_objects":29676,"num_object_clones":0,"num_object_copies":118704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155402,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2801984,"kb_used_data":475356,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374685376,"statfs":{"total":386547056640,"available":383677825024,"internally_reserved":0,"allocated":486764544,"data_stored":486072136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2382349874},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,6,7,8,17,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18694144,"num_objects":-2282,"num_object_clones":0,"num_object_copies":-9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004253"},"pg_stats":[{"pgid":"1.f","version":"35'10534","reported_seq":10586,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.123293+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:39.123293+0000","last_peered":"2024-09-15T21:12:39.123293+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:39.123293+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14893056,"num_objects":1818,"num_object_clones":0,"num_object_copies":7272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10534,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-458}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10718","reported_seq":15611,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:42.474786+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:42.474786+0000","last_peered":"2024-09-15T21:12:42.474786+0000","last_clean":"2024-09-15T21:12:42.474786+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:42.474786+0000","last_undegraded":"2024-09-15T21:12:42.474786+0000","last_fullsized":"2024-09-15T21:12:42.474786+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10532","reported_seq":10618,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.235292+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:40.235292+0000","last_peered":"2024-09-15T21:12:40.235292+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:40.235292+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15563422,"num_objects":1901,"num_object_clones":0,"num_object_copies":7604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1901,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10531,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-357}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10683","reported_seq":10757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.459782+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:42.459782+0000","last_peered":"2024-09-15T21:12:42.459782+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:42.459782+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14521612,"num_objects":1773,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10780","reported_seq":10842,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.035143+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:39.035143+0000","last_peered":"2024-09-15T21:12:39.035143+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:39.035143+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15389964,"num_objects":1879,"num_object_clones":0,"num_object_copies":7516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-432}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10559","reported_seq":10608,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.190023+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:39.190023+0000","last_peered":"2024-09-15T21:12:39.190023+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:39.190023+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14884864,"num_objects":1817,"num_object_clones":0,"num_object_copies":7268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10897","reported_seq":10973,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.450012+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:42.450012+0000","last_peered":"2024-09-15T21:12:42.450012+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:42.450012+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15357196,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10895,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-491}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10935","reported_seq":11010,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.496946+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:42.496946+0000","last_peered":"2024-09-15T21:12:42.496946+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:42.496946+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955788,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10933,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10587","reported_seq":10653,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.045018+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:39.045018+0000","last_peered":"2024-09-15T21:12:39.045018+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:39.045018+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15272472,"num_objects":1865,"num_object_clones":0,"num_object_copies":7460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10584,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-467}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10680","reported_seq":10767,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.346778+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:40.346778+0000","last_peered":"2024-09-15T21:12:40.346778+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:40.346778+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15414540,"num_objects":1882,"num_object_clones":0,"num_object_copies":7528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-453}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10519","reported_seq":15188,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:39.035011+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:39.035011+0000","last_peered":"2024-09-15T21:12:39.035011+0000","last_clean":"2024-09-15T21:12:39.035011+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:39.035011+0000","last_undegraded":"2024-09-15T21:12:39.035011+0000","last_fullsized":"2024-09-15T21:12:39.035011+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15081472,"num_objects":1841,"num_object_clones":0,"num_object_copies":7364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10524","reported_seq":10570,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.121943+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:39.121943+0000","last_peered":"2024-09-15T21:12:39.121943+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:39.121943+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15587974,"num_objects":1903,"num_object_clones":0,"num_object_copies":7612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10523,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-353}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10635","reported_seq":13226,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.389209+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:40.389209+0000","last_peered":"2024-09-15T21:12:40.389209+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:40.389209+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15284870,"num_objects":1866,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10634,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":831}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10624","reported_seq":10687,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:39.039359+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:39.039359+0000","last_peered":"2024-09-15T21:12:39.039359+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:39.039359+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14808332,"num_objects":1808,"num_object_clones":0,"num_object_copies":7232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-471}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10720","reported_seq":10806,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:40.113713+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:40.113713+0000","last_peered":"2024-09-15T21:12:40.113713+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:40.113713+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16061708,"num_objects":1961,"num_object_clones":0,"num_object_copies":7844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-320}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10720","reported_seq":10795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.560538+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:42.560538+0000","last_peered":"2024-09-15T21:12:42.560538+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:42.560538+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980364,"num_objects":1829,"num_object_clones":0,"num_object_copies":7316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":243065378,"num_objects":29676,"num_object_clones":0,"num_object_copies":118704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29676,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":155402,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1274,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":484184064,"data_stored":484184064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160747,"ondisk_log_size":160747,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729444,"kb_used_data":120212,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93642396,"statfs":{"total":96636764160,"available":95889813504,"internally_reserved":0,"allocated":123097088,"data_stored":122923986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732312,"kb_used_data":119432,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93639528,"statfs":{"total":96636764160,"available":95886876672,"internally_reserved":0,"allocated":122298368,"data_stored":122125266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586532,"kb_used_data":117552,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93785308,"statfs":{"total":96636764160,"available":96036155392,"internally_reserved":0,"allocated":120373248,"data_stored":120200146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,7,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753696,"kb_used_data":118160,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93618144,"statfs":{"total":96636764160,"available":95864979456,"internally_reserved":0,"allocated":120995840,"data_stored":120822738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119996416,"data_stored":119996416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":121597952,"data_stored":121597952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122568704,"data_stored":122568704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:46.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:46.670 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:46.670 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:46.701 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-15T21:12:46.150768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237609506,"num_objects":29010,"num_object_clones":0,"num_object_copies":116040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156068,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161013,"ondisk_log_size":161013,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802600,"kb_used_data":467648,"kb_used_omap":14,"kb_used_meta":2334833,"kb_avail":374684760,"statfs":{"total":386547056640,"available":383677194240,"internally_reserved":0,"allocated":478871552,"data_stored":478179144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2390869554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,6,9,12,17,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18292736,"num_objects":-2233,"num_object_clones":0,"num_object_copies":-8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003991"},"pg_stats":[{"pgid":"1.f","version":"35'10600","reported_seq":10652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.234436+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:44.234436+0000","last_peered":"2024-09-15T21:12:44.234436+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:44.234436+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10718","reported_seq":15611,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:42.474786+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:42.474786+0000","last_peered":"2024-09-15T21:12:42.474786+0000","last_clean":"2024-09-15T21:12:42.474786+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:42.474786+0000","last_undegraded":"2024-09-15T21:12:42.474786+0000","last_fullsized":"2024-09-15T21:12:42.474786+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10587","reported_seq":10673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.970670+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:44.970670+0000","last_peered":"2024-09-15T21:12:44.970670+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:44.970670+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15112862,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10586,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10683","reported_seq":10757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.459782+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:42.459782+0000","last_peered":"2024-09-15T21:12:42.459782+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:42.459782+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14521612,"num_objects":1773,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10846","reported_seq":10908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.246787+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:44.246787+0000","last_peered":"2024-09-15T21:12:44.246787+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:44.246787+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14849292,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10615","reported_seq":10666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.129612+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:44.129612+0000","last_peered":"2024-09-15T21:12:44.129612+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:44.129612+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14426112,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10897","reported_seq":10973,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.450012+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:42.450012+0000","last_peered":"2024-09-15T21:12:42.450012+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:42.450012+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15357196,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10895,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-491}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10935","reported_seq":11010,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.496946+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:42.496946+0000","last_peered":"2024-09-15T21:12:42.496946+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:42.496946+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955788,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10933,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10642","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.085686+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:44.085686+0000","last_peered":"2024-09-15T21:12:44.085686+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:44.085686+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14821912,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10639,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10737","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.313536+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:45.313536+0000","last_peered":"2024-09-15T21:12:45.313536+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:45.313536+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947596,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10578","reported_seq":15247,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:44.241048+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:44.241048+0000","last_peered":"2024-09-15T21:12:44.241048+0000","last_clean":"2024-09-15T21:12:44.241048+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:44.241048+0000","last_undegraded":"2024-09-15T21:12:44.241048+0000","last_fullsized":"2024-09-15T21:12:44.241048+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14598144,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10591","reported_seq":10636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.187986+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:44.187986+0000","last_peered":"2024-09-15T21:12:44.187986+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:44.187986+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15039110,"num_objects":1836,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-420}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10697","reported_seq":13415,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.342380+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:45.342380+0000","last_peered":"2024-09-15T21:12:45.342380+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:45.342380+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14776966,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10682","reported_seq":10745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:43.923178+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:43.923178+0000","last_peered":"2024-09-15T21:12:43.923178+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:43.923178+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14333196,"num_objects":1750,"num_object_clones":0,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-529}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10785","reported_seq":10874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.316055+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:45.316055+0000","last_peered":"2024-09-15T21:12:45.316055+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:45.316055+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529228,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-385}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10720","reported_seq":10795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.560538+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:42.560538+0000","last_peered":"2024-09-15T21:12:42.560538+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:42.560538+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980364,"num_objects":1829,"num_object_clones":0,"num_object_copies":7316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237609506,"num_objects":29010,"num_object_clones":0,"num_object_copies":116040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156068,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476168192,"data_stored":476168192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161013,"ondisk_log_size":161013,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729696,"kb_used_data":116300,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93642144,"statfs":{"total":96636764160,"available":95889555456,"internally_reserved":0,"allocated":119091200,"data_stored":118918098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1,2,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732676,"kb_used_data":115636,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93639164,"statfs":{"total":96636764160,"available":95886503936,"internally_reserved":0,"allocated":118411264,"data_stored":118238162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586532,"kb_used_data":117552,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93785308,"statfs":{"total":96636764160,"available":96036155392,"internally_reserved":0,"allocated":120373248,"data_stored":120200146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,7,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753696,"kb_used_data":118160,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93618144,"statfs":{"total":96636764160,"available":95864979456,"internally_reserved":0,"allocated":120995840,"data_stored":120822738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119996416,"data_stored":119996416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117620736,"data_stored":117620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118530048,"data_stored":118530048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10600", 'reported_seq': 10652, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.234436+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:44.234436+0000', 'last_peered': '2024-09-15T21:12:44.234436+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:44.234436+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14352384, 'num_objects': 1752, 'num_object_clones': 0, 'num_object_copies': 7008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1752, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10600, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -524}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10587", 'reported_seq': 10673, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.970670+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:44.970670+0000', 'last_peered': '2024-09-15T21:12:44.970670+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:44.970670+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15112862, 'num_objects': 1846, 'num_object_clones': 0, 'num_object_copies': 7384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1846, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10586, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -412}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10683", 'reported_seq': 10757, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:42.459782+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:42.459782+0000', 'last_peered': '2024-09-15T21:12:42.459782+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:42.459782+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14521612, 'num_objects': 1773, 'num_object_clones': 0, 'num_object_copies': 7092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10681, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -561}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:46.702 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10846", 'reported_seq': 10908, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.246787+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:44.246787+0000', 'last_peered': '2024-09-15T21:12:44.246787+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:44.246787+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14849292, 'num_objects': 1813, 'num_object_clones': 0, 'num_object_copies': 7252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10844, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -498}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10615", 'reported_seq': 10666, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.129612+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:44.129612+0000', 'last_peered': '2024-09-15T21:12:44.129612+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:44.129612+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14426112, 'num_objects': 1761, 'num_object_clones': 0, 'num_object_copies': 7044, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1761, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10615, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -555}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10897", 'reported_seq': 10973, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:42.450012+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:42.450012+0000', 'last_peered': '2024-09-15T21:12:42.450012+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:42.450012+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15357196, 'num_objects': 1875, 'num_object_clones': 0, 'num_object_copies': 7500, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1875, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10895, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -491}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10935", 'reported_seq': 11010, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:42.496946+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:42.496946+0000', 'last_peered': '2024-09-15T21:12:42.496946+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:42.496946+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14955788, 'num_objects': 1826, 'num_object_clones': 0, 'num_object_copies': 7304, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1826, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10933, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -648}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10642", 'reported_seq': 10708, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.085686+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:44.085686+0000', 'last_peered': '2024-09-15T21:12:44.085686+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:44.085686+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14821912, 'num_objects': 1810, 'num_object_clones': 0, 'num_object_copies': 7240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1810, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10639, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -522}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:46.703 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10737", 'reported_seq': 10824, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:45.313536+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:45.313536+0000', 'last_peered': '2024-09-15T21:12:45.313536+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:45.313536+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10037, 'log_dups_size': 0, 'ondisk_log_size': 10037, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14947596, 'num_objects': 1825, 'num_object_clones': 0, 'num_object_copies': 7300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1825, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10735, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -510}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10591", 'reported_seq': 10636, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:44.187986+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:44.187986+0000', 'last_peered': '2024-09-15T21:12:44.187986+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:44.187986+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'500", 'ondisk_log_start': "27'500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15039110, 'num_objects': 1836, 'num_object_clones': 0, 'num_object_copies': 7344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1836, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10590, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -420}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10697", 'reported_seq': 13415, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:45.342380+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:45.342380+0000', 'last_peered': '2024-09-15T21:12:45.342380+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:45.342380+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14776966, 'num_objects': 1804, 'num_object_clones': 0, 'num_object_copies': 7216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 971, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10696, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1317, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10682", 'reported_seq': 10745, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:43.923178+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:43.923178+0000', 'last_peered': '2024-09-15T21:12:43.923178+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:43.923178+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14333196, 'num_objects': 1750, 'num_object_clones': 0, 'num_object_copies': 7000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1750, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10680, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -529}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10785", 'reported_seq': 10874, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:45.316055+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:45.316055+0000', 'last_peered': '2024-09-15T21:12:45.316055+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:45.316055+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10085, 'log_dups_size': 0, 'ondisk_log_size': 10085, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 15529228, 'num_objects': 1896, 'num_object_clones': 0, 'num_object_copies': 7584, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1896, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10783, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -385}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:46.704 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10720", 'reported_seq': 10795, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:42.560538+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:42.560538+0000', 'last_peered': '2024-09-15T21:12:42.560538+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:42.560538+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14980364, 'num_objects': 1829, 'num_object_clones': 0, 'num_object_copies': 7316, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1829, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10718, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -486}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:46.705 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:47.478 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:47.478 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31125/116040 objects degraded (26.823%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":633,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":29010,"data_bytes":237609506,"bytes_used":2869862400,"bytes_avail":383677194240,"bytes_total":386547056640,"degraded_objects":31125,"degraded_total":116040,"degraded_ratio":0.26822647362978286,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:47.500 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:47.501 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:48.073 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:48.073 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:48.094 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-09-15T21:12:46.150768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":237609506,"num_objects":29010,"num_object_clones":0,"num_object_copies":116040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156068,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161013,"ondisk_log_size":161013,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2802600,"kb_used_data":467648,"kb_used_omap":14,"kb_used_meta":2334833,"kb_avail":374684760,"statfs":{"total":386547056640,"available":383677194240,"internally_reserved":0,"allocated":478871552,"data_stored":478179144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2390869554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,6,9,12,17,11,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18292736,"num_objects":-2233,"num_object_clones":0,"num_object_copies":-8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":177,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003991"},"pg_stats":[{"pgid":"1.f","version":"35'10600","reported_seq":10652,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.234436+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:44.234436+0000","last_peered":"2024-09-15T21:12:44.234436+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:44.234436+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":1752,"num_object_clones":0,"num_object_copies":7008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10600,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-524}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10718","reported_seq":15611,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:42.474786+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:42.474786+0000","last_peered":"2024-09-15T21:12:42.474786+0000","last_clean":"2024-09-15T21:12:42.474786+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:42.474786+0000","last_undegraded":"2024-09-15T21:12:42.474786+0000","last_fullsized":"2024-09-15T21:12:42.474786+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15007744,"num_objects":1832,"num_object_clones":0,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1384,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10587","reported_seq":10673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.970670+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:44.970670+0000","last_peered":"2024-09-15T21:12:44.970670+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:44.970670+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15112862,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10586,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10683","reported_seq":10757,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.459782+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:42.459782+0000","last_peered":"2024-09-15T21:12:42.459782+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:42.459782+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14521612,"num_objects":1773,"num_object_clones":0,"num_object_copies":7092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-561}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10846","reported_seq":10908,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.246787+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:44.246787+0000","last_peered":"2024-09-15T21:12:44.246787+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:44.246787+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14849292,"num_objects":1813,"num_object_clones":0,"num_object_copies":7252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10615","reported_seq":10666,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.129612+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:44.129612+0000","last_peered":"2024-09-15T21:12:44.129612+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:44.129612+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14426112,"num_objects":1761,"num_object_clones":0,"num_object_copies":7044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-555}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10897","reported_seq":10973,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.450012+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:42.450012+0000","last_peered":"2024-09-15T21:12:42.450012+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:42.450012+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15357196,"num_objects":1875,"num_object_clones":0,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10895,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-491}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10935","reported_seq":11010,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.496946+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:42.496946+0000","last_peered":"2024-09-15T21:12:42.496946+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:42.496946+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14955788,"num_objects":1826,"num_object_clones":0,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10933,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-648}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10642","reported_seq":10708,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.085686+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:44.085686+0000","last_peered":"2024-09-15T21:12:44.085686+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:44.085686+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14821912,"num_objects":1810,"num_object_clones":0,"num_object_copies":7240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10639,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-522}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10737","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.313536+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:45.313536+0000","last_peered":"2024-09-15T21:12:45.313536+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:45.313536+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947596,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10578","reported_seq":15247,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:44.241048+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:44.241048+0000","last_peered":"2024-09-15T21:12:44.241048+0000","last_clean":"2024-09-15T21:12:44.241048+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:44.241048+0000","last_undegraded":"2024-09-15T21:12:44.241048+0000","last_fullsized":"2024-09-15T21:12:44.241048+0000","mapping_epoch":32,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14598144,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4689,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10591","reported_seq":10636,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.187986+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:44.187986+0000","last_peered":"2024-09-15T21:12:44.187986+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:44.187986+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15039110,"num_objects":1836,"num_object_clones":0,"num_object_copies":7344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1836,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10590,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-420}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10697","reported_seq":13415,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.342380+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:45.342380+0000","last_peered":"2024-09-15T21:12:45.342380+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:45.342380+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14776966,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10682","reported_seq":10745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:43.923178+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:43.923178+0000","last_peered":"2024-09-15T21:12:43.923178+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:43.923178+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14333196,"num_objects":1750,"num_object_clones":0,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-529}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10785","reported_seq":10874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.316055+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:45.316055+0000","last_peered":"2024-09-15T21:12:45.316055+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:45.316055+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529228,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-385}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10720","reported_seq":10795,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:42.560538+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:42.560538+0000","last_peered":"2024-09-15T21:12:42.560538+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:42.560538+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14980364,"num_objects":1829,"num_object_clones":0,"num_object_copies":7316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10718,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-486}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":237609506,"num_objects":29010,"num_object_clones":0,"num_object_copies":116040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156068,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":476168192,"data_stored":476168192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161013,"ondisk_log_size":161013,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729696,"kb_used_data":116300,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93642144,"statfs":{"total":96636764160,"available":95889555456,"internally_reserved":0,"allocated":119091200,"data_stored":118918098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,3,1,2,4,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732676,"kb_used_data":115636,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93639164,"statfs":{"total":96636764160,"available":95886503936,"internally_reserved":0,"allocated":118411264,"data_stored":118238162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051689,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586532,"kb_used_data":117552,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93785308,"statfs":{"total":96636764160,"available":96036155392,"internally_reserved":0,"allocated":120373248,"data_stored":120200146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5,7,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247927,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":753696,"kb_used_data":118160,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93618144,"statfs":{"total":96636764160,"available":95864979456,"internally_reserved":0,"allocated":120995840,"data_stored":120822738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,1,3,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":119996416,"data_stored":119996416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":120020992,"data_stored":120020992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117620736,"data_stored":117620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":118530048,"data_stored":118530048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:51.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:51.726 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:51.726 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:51.750 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-15T21:12:50.152075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":231359010,"num_objects":28247,"num_object_clones":0,"num_object_copies":112988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156831,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160976,"ondisk_log_size":160976,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2790204,"kb_used_data":455276,"kb_used_omap":14,"kb_used_meta":2334833,"kb_avail":374697156,"statfs":{"total":386547056640,"available":383689887744,"internally_reserved":0,"allocated":466202624,"data_stored":465510216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2390869554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,10,2,18,11,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19914752,"num_objects":-2431,"num_object_clones":0,"num_object_copies":-9724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2431,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003918"},"pg_stats":[{"pgid":"1.f","version":"35'10665","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.382931+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:49.382931+0000","last_peered":"2024-09-15T21:12:49.382931+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:49.382931+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10786","reported_seq":15679,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:47.671684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:47.671684+0000","last_peered":"2024-09-15T21:12:47.671684+0000","last_clean":"2024-09-15T21:12:47.671684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:47.671684+0000","last_undegraded":"2024-09-15T21:12:47.671684+0000","last_fullsized":"2024-09-15T21:12:47.671684+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14450688,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10587","reported_seq":10673,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:44.970670+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:44.970670+0000","last_peered":"2024-09-15T21:12:44.970670+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:44.970670+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'500","ondisk_log_start":"27'500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15112862,"num_objects":1846,"num_object_clones":0,"num_object_copies":7384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1846,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10586,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-412}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10727","reported_seq":10802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.161825+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:47.161825+0000","last_peered":"2024-09-15T21:12:47.161825+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:47.161825+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14161164,"num_objects":1729,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10918","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.358786+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:49.358786+0000","last_peered":"2024-09-15T21:12:49.358786+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:49.358786+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259468,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10678","reported_seq":10729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.284445+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:49.284445+0000","last_peered":"2024-09-15T21:12:49.284445+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:49.284445+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13910016,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10971","reported_seq":11046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.579446+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:47.579446+0000","last_peered":"2024-09-15T21:12:47.579446+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:47.579446+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10999","reported_seq":11074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.337365+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:47.337365+0000","last_peered":"2024-09-15T21:12:47.337365+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:47.337365+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10717","reported_seq":10781,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.445923+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:49.445923+0000","last_peered":"2024-09-15T21:12:49.445923+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:49.445923+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14207512,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10737","reported_seq":10824,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.313536+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:45.313536+0000","last_peered":"2024-09-15T21:12:45.313536+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:45.313536+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14947596,"num_objects":1825,"num_object_clones":0,"num_object_copies":7300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-510}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10651","reported_seq":15320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:49.408401+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:49.408401+0000","last_peered":"2024-09-15T21:12:49.408401+0000","last_clean":"2024-09-15T21:12:49.408401+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:49.408401+0000","last_undegraded":"2024-09-15T21:12:49.408401+0000","last_fullsized":"2024-09-15T21:12:49.408401+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14000128,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4762,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10645","reported_seq":10691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.320720+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:49.320720+0000","last_peered":"2024-09-15T21:12:49.320720+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:49.320720+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596742,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10697","reported_seq":13415,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.342380+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:45.342380+0000","last_peered":"2024-09-15T21:12:45.342380+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:45.342380+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14776966,"num_objects":1804,"num_object_clones":0,"num_object_copies":7216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":971,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10734","reported_seq":10797,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.209854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:49.209854+0000","last_peered":"2024-09-15T21:12:49.209854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:49.209854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13907212,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10785","reported_seq":10874,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:45.316055+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:45.316055+0000","last_peered":"2024-09-15T21:12:45.316055+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:45.316055+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15529228,"num_objects":1896,"num_object_clones":0,"num_object_copies":7584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10783,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-385}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10779","reported_seq":10854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.356936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:47.356936+0000","last_peered":"2024-09-15T21:12:47.356936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:47.356936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14497036,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":231359010,"num_objects":28247,"num_object_clones":0,"num_object_copies":112988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":156831,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":463917056,"data_stored":463917056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160976,"ondisk_log_size":160976,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729980,"kb_used_data":112428,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93641860,"statfs":{"total":96636764160,"available":95889264640,"internally_reserved":0,"allocated":115126272,"data_stored":114953170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,9,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732676,"kb_used_data":115636,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93639164,"statfs":{"total":96636764160,"available":95886503936,"internally_reserved":0,"allocated":118411264,"data_stored":118238162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,2,5,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586720,"kb_used_data":113616,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93785120,"statfs":{"total":96636764160,"available":96035962880,"internally_reserved":0,"allocated":116342784,"data_stored":116169682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":113596,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":116322304,"data_stored":116149202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115941376,"data_stored":115941376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117620736,"data_stored":117620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114470912,"data_stored":114470912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:51.751 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:52.404 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:52.405 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:52.437 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-15T21:12:52.152960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229401122,"num_objects":28008,"num_object_clones":0,"num_object_copies":112032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157070,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773836,"kb_used_data":451376,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374713524,"statfs":{"total":386547056640,"available":383706648576,"internally_reserved":0,"allocated":462209024,"data_stored":461516616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2378090034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,3,19,12,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18415616,"num_objects":-2248,"num_object_clones":0,"num_object_copies":-8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004156"},"pg_stats":[{"pgid":"1.f","version":"35'10665","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.382931+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:49.382931+0000","last_peered":"2024-09-15T21:12:49.382931+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:49.382931+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10786","reported_seq":15679,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:47.671684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:47.671684+0000","last_peered":"2024-09-15T21:12:47.671684+0000","last_clean":"2024-09-15T21:12:47.671684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:47.671684+0000","last_undegraded":"2024-09-15T21:12:47.671684+0000","last_fullsized":"2024-09-15T21:12:47.671684+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14450688,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10636","reported_seq":10721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.333640+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:50.333640+0000","last_peered":"2024-09-15T21:12:50.333640+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:50.333640+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14711454,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10635,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10727","reported_seq":10802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.161825+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:47.161825+0000","last_peered":"2024-09-15T21:12:47.161825+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:47.161825+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14161164,"num_objects":1729,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10918","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.358786+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:49.358786+0000","last_peered":"2024-09-15T21:12:49.358786+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:49.358786+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259468,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10678","reported_seq":10729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.284445+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:49.284445+0000","last_peered":"2024-09-15T21:12:49.284445+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:49.284445+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13910016,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10971","reported_seq":11046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.579446+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:47.579446+0000","last_peered":"2024-09-15T21:12:47.579446+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:47.579446+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10999","reported_seq":11074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.337365+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:47.337365+0000","last_peered":"2024-09-15T21:12:47.337365+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:47.337365+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10717","reported_seq":10781,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.445923+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:49.445923+0000","last_peered":"2024-09-15T21:12:49.445923+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:49.445923+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14207512,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10793","reported_seq":10879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.426486+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:50.426486+0000","last_peered":"2024-09-15T21:12:50.426486+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:50.426486+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488844,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10651","reported_seq":15320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:49.408401+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:49.408401+0000","last_peered":"2024-09-15T21:12:49.408401+0000","last_clean":"2024-09-15T21:12:49.408401+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:49.408401+0000","last_undegraded":"2024-09-15T21:12:49.408401+0000","last_fullsized":"2024-09-15T21:12:49.408401+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14000128,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4762,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10645","reported_seq":10691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.320720+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:49.320720+0000","last_peered":"2024-09-15T21:12:49.320720+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:49.320720+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596742,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10757","reported_seq":13595,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.454656+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:50.454656+0000","last_peered":"2024-09-15T21:12:50.454656+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:50.454656+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14285446,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10756,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1377,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10734","reported_seq":10797,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.209854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:49.209854+0000","last_peered":"2024-09-15T21:12:49.209854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:49.209854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13907212,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10859","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.394196+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:50.394196+0000","last_peered":"2024-09-15T21:12:50.394196+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:50.394196+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14923020,"num_objects":1822,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-459}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10779","reported_seq":10854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.356936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:47.356936+0000","last_peered":"2024-09-15T21:12:47.356936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:47.356936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14497036,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229401122,"num_objects":28008,"num_object_clones":0,"num_object_copies":112032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157070,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459948032,"data_stored":459948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729980,"kb_used_data":112428,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93641860,"statfs":{"total":96636764160,"available":95889264640,"internally_reserved":0,"allocated":115126272,"data_stored":114953170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,9,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716308,"kb_used_data":111736,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93655532,"statfs":{"total":96636764160,"available":95903264768,"internally_reserved":0,"allocated":114417664,"data_stored":114244562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586720,"kb_used_data":113616,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93785120,"statfs":{"total":96636764160,"available":96035962880,"internally_reserved":0,"allocated":116342784,"data_stored":116169682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":113596,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":116322304,"data_stored":116149202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115941376,"data_stored":115941376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113651712,"data_stored":113651712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114470912,"data_stored":114470912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:52.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:52.438 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10665", 'reported_seq': 10715, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.382931+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:49.382931+0000', 'last_peered': '2024-09-15T21:12:49.382931+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:49.382931+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13819904, 'num_objects': 1687, 'num_object_clones': 0, 'num_object_copies': 6748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1687, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10665, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -589}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.438 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10636", 'reported_seq': 10721, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:50.333640+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:50.333640+0000', 'last_peered': '2024-09-15T21:12:50.333640+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:50.333640+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14711454, 'num_objects': 1797, 'num_object_clones': 0, 'num_object_copies': 7188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1797, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10635, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -461}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10727", 'reported_seq': 10802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:47.161825+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:47.161825+0000', 'last_peered': '2024-09-15T21:12:47.161825+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:47.161825+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14161164, 'num_objects': 1729, 'num_object_clones': 0, 'num_object_copies': 6916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1729, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10725, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -605}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10918", 'reported_seq': 10980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.358786+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:49.358786+0000', 'last_peered': '2024-09-15T21:12:49.358786+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:49.358786+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14259468, 'num_objects': 1741, 'num_object_clones': 0, 'num_object_copies': 6964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1741, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10916, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -570}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10678", 'reported_seq': 10729, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.284445+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:49.284445+0000', 'last_peered': '2024-09-15T21:12:49.284445+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:49.284445+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13910016, 'num_objects': 1698, 'num_object_clones': 0, 'num_object_copies': 6792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10678, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -618}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'10971", 'reported_seq': 11046, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:47.579446+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:47.579446+0000', 'last_peered': '2024-09-15T21:12:47.579446+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:47.579446+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14750988, 'num_objects': 1801, 'num_object_clones': 0, 'num_object_copies': 7204, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1801, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10969, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -565}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:52.439 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'10999", 'reported_seq': 11074, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:47.337365+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:47.337365+0000', 'last_peered': '2024-09-15T21:12:47.337365+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:47.337365+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14431500, 'num_objects': 1762, 'num_object_clones': 0, 'num_object_copies': 7048, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1762, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10997, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -712}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10717", 'reported_seq': 10781, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.445923+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:49.445923+0000', 'last_peered': '2024-09-15T21:12:49.445923+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:49.445923+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14207512, 'num_objects': 1735, 'num_object_clones': 0, 'num_object_copies': 6940, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1735, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10714, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -597}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10793", 'reported_seq': 10879, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:50.426486+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:50.426486+0000', 'last_peered': '2024-09-15T21:12:50.426486+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:50.426486+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14488844, 'num_objects': 1769, 'num_object_clones': 0, 'num_object_copies': 7076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10791, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -566}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10645", 'reported_seq': 10691, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.320720+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:49.320720+0000', 'last_peered': '2024-09-15T21:12:49.320720+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:49.320720+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14596742, 'num_objects': 1782, 'num_object_clones': 0, 'num_object_copies': 7128, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1782, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10644, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -474}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.440 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10757", 'reported_seq': 13595, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:50.454656+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:50.454656+0000', 'last_peered': '2024-09-15T21:12:50.454656+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:50.454656+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14285446, 'num_objects': 1744, 'num_object_clones': 0, 'num_object_copies': 6976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 911, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1744, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10756, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1377, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 833}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10734", 'reported_seq': 10797, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:49.209854+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:49.209854+0000', 'last_peered': '2024-09-15T21:12:49.209854+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:49.209854+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13907212, 'num_objects': 1698, 'num_object_clones': 0, 'num_object_copies': 6792, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1698, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10732, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -581}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10859", 'reported_seq': 10948, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:50.394196+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:50.394196+0000', 'last_peered': '2024-09-15T21:12:50.394196+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:50.394196+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14923020, 'num_objects': 1822, 'num_object_clones': 0, 'num_object_copies': 7288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1822, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10857, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -459}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:52.441 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10779", 'reported_seq': 10854, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:47.356936+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:47.356936+0000', 'last_peered': '2024-09-15T21:12:47.356936+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:47.356936+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14497036, 'num_objects': 1770, 'num_object_clones': 0, 'num_object_copies': 7080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10777, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -545}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:52.441 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:53.233 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:53.233 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31065/112032 objects degraded (27.729%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":638,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":28008,"data_bytes":229401122,"bytes_used":2840408064,"bytes_avail":383706648576,"bytes_total":386547056640,"degraded_objects":31065,"degraded_total":112032,"degraded_ratio":0.27728684661525277,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":187},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:53.270 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:53.270 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:53.836 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:53.836 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:53.860 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-15T21:12:52.152960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":229401122,"num_objects":28008,"num_object_clones":0,"num_object_copies":112032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157070,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2773836,"kb_used_data":451376,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374713524,"statfs":{"total":386547056640,"available":383706648576,"internally_reserved":0,"allocated":462209024,"data_stored":461516616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2378090034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,6,3,19,12,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18415616,"num_objects":-2248,"num_object_clones":0,"num_object_copies":-8992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-176,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004156"},"pg_stats":[{"pgid":"1.f","version":"35'10665","reported_seq":10715,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.382931+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:49.382931+0000","last_peered":"2024-09-15T21:12:49.382931+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:49.382931+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13819904,"num_objects":1687,"num_object_clones":0,"num_object_copies":6748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10665,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10786","reported_seq":15679,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:47.671684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:47.671684+0000","last_peered":"2024-09-15T21:12:47.671684+0000","last_clean":"2024-09-15T21:12:47.671684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:47.671684+0000","last_undegraded":"2024-09-15T21:12:47.671684+0000","last_fullsized":"2024-09-15T21:12:47.671684+0000","mapping_epoch":34,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14450688,"num_objects":1764,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1452,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10636","reported_seq":10721,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.333640+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:50.333640+0000","last_peered":"2024-09-15T21:12:50.333640+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:50.333640+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14711454,"num_objects":1797,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10635,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-461}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10727","reported_seq":10802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.161825+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:47.161825+0000","last_peered":"2024-09-15T21:12:47.161825+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:47.161825+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14161164,"num_objects":1729,"num_object_clones":0,"num_object_copies":6916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10725,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-605}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10918","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.358786+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:49.358786+0000","last_peered":"2024-09-15T21:12:49.358786+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:49.358786+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14259468,"num_objects":1741,"num_object_clones":0,"num_object_copies":6964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-570}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10678","reported_seq":10729,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.284445+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:49.284445+0000","last_peered":"2024-09-15T21:12:49.284445+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:49.284445+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13910016,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-618}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'10971","reported_seq":11046,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.579446+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:47.579446+0000","last_peered":"2024-09-15T21:12:47.579446+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:47.579446+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14750988,"num_objects":1801,"num_object_clones":0,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10969,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-565}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'10999","reported_seq":11074,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.337365+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:47.337365+0000","last_peered":"2024-09-15T21:12:47.337365+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:47.337365+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14431500,"num_objects":1762,"num_object_clones":0,"num_object_copies":7048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-712}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10717","reported_seq":10781,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.445923+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:49.445923+0000","last_peered":"2024-09-15T21:12:49.445923+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:49.445923+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14207512,"num_objects":1735,"num_object_clones":0,"num_object_copies":6940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10714,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-597}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10793","reported_seq":10879,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.426486+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:50.426486+0000","last_peered":"2024-09-15T21:12:50.426486+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:50.426486+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14488844,"num_objects":1769,"num_object_clones":0,"num_object_copies":7076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-566}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10651","reported_seq":15320,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:49.408401+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:49.408401+0000","last_peered":"2024-09-15T21:12:49.408401+0000","last_clean":"2024-09-15T21:12:49.408401+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:49.408401+0000","last_undegraded":"2024-09-15T21:12:49.408401+0000","last_fullsized":"2024-09-15T21:12:49.408401+0000","mapping_epoch":32,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14000128,"num_objects":1709,"num_object_clones":0,"num_object_copies":6836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4762,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10645","reported_seq":10691,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.320720+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:49.320720+0000","last_peered":"2024-09-15T21:12:49.320720+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:49.320720+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14596742,"num_objects":1782,"num_object_clones":0,"num_object_copies":7128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10644,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-474}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10757","reported_seq":13595,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.454656+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:50.454656+0000","last_peered":"2024-09-15T21:12:50.454656+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:50.454656+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14285446,"num_objects":1744,"num_object_clones":0,"num_object_copies":6976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":911,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10756,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1377,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":833}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10734","reported_seq":10797,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:49.209854+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:49.209854+0000","last_peered":"2024-09-15T21:12:49.209854+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:49.209854+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13907212,"num_objects":1698,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10732,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-581}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10859","reported_seq":10948,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:50.394196+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:50.394196+0000","last_peered":"2024-09-15T21:12:50.394196+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:50.394196+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14923020,"num_objects":1822,"num_object_clones":0,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10857,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-459}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10779","reported_seq":10854,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:47.356936+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:47.356936+0000","last_peered":"2024-09-15T21:12:47.356936+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:47.356936+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14497036,"num_objects":1770,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10777,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-545}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":229401122,"num_objects":28008,"num_object_clones":0,"num_object_copies":112032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31065,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28008,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157070,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":459948032,"data_stored":459948032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160915,"ondisk_log_size":160915,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729980,"kb_used_data":112428,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93641860,"statfs":{"total":96636764160,"available":95889264640,"internally_reserved":0,"allocated":115126272,"data_stored":114953170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,9,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716308,"kb_used_data":111736,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93655532,"statfs":{"total":96636764160,"available":95903264768,"internally_reserved":0,"allocated":114417664,"data_stored":114244562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051690,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":586720,"kb_used_data":113616,"kb_used_omap":4,"kb_used_meta":473083,"kb_avail":93785120,"statfs":{"total":96636764160,"available":96035962880,"internally_reserved":0,"allocated":116342784,"data_stored":116169682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":484437711},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247928,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":740828,"kb_used_data":113596,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93631012,"statfs":{"total":96636764160,"available":95878156288,"internally_reserved":0,"allocated":116322304,"data_stored":116149202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,0,3,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":115941376,"data_stored":115941376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":115884032,"data_stored":115884032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113651712,"data_stored":113651712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":114470912,"data_stored":114470912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:56.861 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:57.432 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:57.433 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:57.455 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-15T21:12:56.154525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":221504034,"num_objects":27044,"num_object_clones":0,"num_object_copies":108176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158034,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741572,"kb_used_data":435772,"kb_used_omap":14,"kb_used_meta":2305713,"kb_avail":374745788,"statfs":{"total":386547056640,"available":383739686912,"internally_reserved":0,"allocated":446230528,"data_stored":445538120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2361050674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,8,19,15,6,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21561344,"num_objects":-2632,"num_object_clones":0,"num_object_copies":-10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004214"},"pg_stats":[{"pgid":"1.f","version":"35'10719","reported_seq":10771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.277425+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:54.277425+0000","last_peered":"2024-09-15T21:12:54.277425+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:54.277425+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13377536,"num_objects":1633,"num_object_clones":0,"num_object_copies":6532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10856","reported_seq":15749,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:52.841101+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:52.841101+0000","last_peered":"2024-09-15T21:12:52.841101+0000","last_clean":"2024-09-15T21:12:52.841101+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:52.841101+0000","last_undegraded":"2024-09-15T21:12:52.841101+0000","last_fullsized":"2024-09-15T21:12:52.841101+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13877248,"num_objects":1694,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10697","reported_seq":10783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.375070+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:55.375070+0000","last_peered":"2024-09-15T21:12:55.375070+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:55.375070+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14211742,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10696,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10789","reported_seq":10861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.438101+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:52.438101+0000","last_peered":"2024-09-15T21:12:52.438101+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:52.438101+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13653260,"num_objects":1667,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10983","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.249328+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:54.249328+0000","last_peered":"2024-09-15T21:12:54.249328+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:54.249328+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10740","reported_seq":10790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.127011+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:54.127011+0000","last_peered":"2024-09-15T21:12:54.127011+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:54.127011+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13402112,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10740,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11024","reported_seq":11100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.539450+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:52.539450+0000","last_peered":"2024-09-15T21:12:52.539450+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:52.539450+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14316812,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11048","reported_seq":11123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.587407+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:52.587407+0000","last_peered":"2024-09-15T21:12:52.587407+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:52.587407+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14030092,"num_objects":1713,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10778","reported_seq":10844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.056377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:54.056377+0000","last_peered":"2024-09-15T21:12:54.056377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:54.056377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13707800,"num_objects":1674,"num_object_clones":0,"num_object_copies":6696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10851","reported_seq":10936,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.196197+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:55.196197+0000","last_peered":"2024-09-15T21:12:55.196197+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:55.196197+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14013708,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-624}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10713","reported_seq":15382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:54.349183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:54.349183+0000","last_peered":"2024-09-15T21:12:54.349183+0000","last_clean":"2024-09-15T21:12:54.349183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:54.349183+0000","last_undegraded":"2024-09-15T21:12:54.349183+0000","last_fullsized":"2024-09-15T21:12:54.349183+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10705","reported_seq":10751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.347218+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:54.347218+0000","last_peered":"2024-09-15T21:12:54.347218+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:54.347218+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105222,"num_objects":1722,"num_object_clones":0,"num_object_copies":6888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10820","reported_seq":13755,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.468418+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:55.468418+0000","last_peered":"2024-09-15T21:12:55.468418+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:55.468418+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769350,"num_objects":1681,"num_object_clones":0,"num_object_copies":6724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10819,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10800","reported_seq":10863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.413358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:54.413358+0000","last_peered":"2024-09-15T21:12:54.413358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:54.413358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13366540,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10798,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10914","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.456883+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:55.456883+0000","last_peered":"2024-09-15T21:12:55.456883+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:55.456883+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14472460,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10842","reported_seq":10916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.431238+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:52.431238+0000","last_peered":"2024-09-15T21:12:52.431238+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:52.431238+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980940,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":221504034,"num_objects":27044,"num_object_clones":0,"num_object_copies":108176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158034,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":443850752,"data_stored":443850752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713660,"kb_used_data":108588,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93658180,"statfs":{"total":96636764160,"available":95905976320,"internally_reserved":0,"allocated":111194112,"data_stored":111021010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,1,2,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716620,"kb_used_data":107892,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93655220,"statfs":{"total":96636764160,"available":95902945280,"internally_reserved":0,"allocated":110481408,"data_stored":110308306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570160,"kb_used_data":109536,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93801680,"statfs":{"total":96636764160,"available":96052920320,"internally_reserved":0,"allocated":112164864,"data_stored":111991762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741132,"kb_used_data":109756,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93630708,"statfs":{"total":96636764160,"available":95877844992,"internally_reserved":0,"allocated":112390144,"data_stored":112217042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,8,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111931392,"data_stored":111931392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111771648,"data_stored":111771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110497792,"data_stored":110497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:57.456 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:58.093 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:58.093 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:58.123 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-15T21:12:56.154525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":221504034,"num_objects":27044,"num_object_clones":0,"num_object_copies":108176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158034,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741572,"kb_used_data":435772,"kb_used_omap":14,"kb_used_meta":2305713,"kb_avail":374745788,"statfs":{"total":386547056640,"available":383739686912,"internally_reserved":0,"allocated":446230528,"data_stored":445538120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2361050674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,8,19,15,6,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21561344,"num_objects":-2632,"num_object_clones":0,"num_object_copies":-10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2632,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004214"},"pg_stats":[{"pgid":"1.f","version":"35'10719","reported_seq":10771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.277425+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:54.277425+0000","last_peered":"2024-09-15T21:12:54.277425+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:54.277425+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13377536,"num_objects":1633,"num_object_clones":0,"num_object_copies":6532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10856","reported_seq":15749,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:52.841101+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:52.841101+0000","last_peered":"2024-09-15T21:12:52.841101+0000","last_clean":"2024-09-15T21:12:52.841101+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:52.841101+0000","last_undegraded":"2024-09-15T21:12:52.841101+0000","last_fullsized":"2024-09-15T21:12:52.841101+0000","mapping_epoch":34,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13877248,"num_objects":1694,"num_object_clones":0,"num_object_copies":6776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1522,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10697","reported_seq":10783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.375070+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:55.375070+0000","last_peered":"2024-09-15T21:12:55.375070+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:55.375070+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14211742,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10696,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10789","reported_seq":10861,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.438101+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:52.438101+0000","last_peered":"2024-09-15T21:12:52.438101+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:52.438101+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13653260,"num_objects":1667,"num_object_clones":0,"num_object_copies":6668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-667}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10983","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.249328+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:54.249328+0000","last_peered":"2024-09-15T21:12:54.249328+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:54.249328+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10740","reported_seq":10790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.127011+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:54.127011+0000","last_peered":"2024-09-15T21:12:54.127011+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:54.127011+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13402112,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10740,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11024","reported_seq":11100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.539450+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:52.539450+0000","last_peered":"2024-09-15T21:12:52.539450+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:52.539450+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14316812,"num_objects":1748,"num_object_clones":0,"num_object_copies":6992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11022,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-618}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11048","reported_seq":11123,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.587407+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:52.587407+0000","last_peered":"2024-09-15T21:12:52.587407+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:52.587407+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14030092,"num_objects":1713,"num_object_clones":0,"num_object_copies":6852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-761}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10778","reported_seq":10844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.056377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:54.056377+0000","last_peered":"2024-09-15T21:12:54.056377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:54.056377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13707800,"num_objects":1674,"num_object_clones":0,"num_object_copies":6696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10851","reported_seq":10936,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.196197+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:55.196197+0000","last_peered":"2024-09-15T21:12:55.196197+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:55.196197+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14013708,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-624}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10713","reported_seq":15382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:54.349183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:54.349183+0000","last_peered":"2024-09-15T21:12:54.349183+0000","last_clean":"2024-09-15T21:12:54.349183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:54.349183+0000","last_undegraded":"2024-09-15T21:12:54.349183+0000","last_fullsized":"2024-09-15T21:12:54.349183+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10705","reported_seq":10751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.347218+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:54.347218+0000","last_peered":"2024-09-15T21:12:54.347218+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:54.347218+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105222,"num_objects":1722,"num_object_clones":0,"num_object_copies":6888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10820","reported_seq":13755,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.468418+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:55.468418+0000","last_peered":"2024-09-15T21:12:55.468418+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:55.468418+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769350,"num_objects":1681,"num_object_clones":0,"num_object_copies":6724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10819,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10800","reported_seq":10863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.413358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:54.413358+0000","last_peered":"2024-09-15T21:12:54.413358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:54.413358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13366540,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10798,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10914","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.456883+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:55.456883+0000","last_peered":"2024-09-15T21:12:55.456883+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:55.456883+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14472460,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10842","reported_seq":10916,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:52.431238+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:52.431238+0000","last_peered":"2024-09-15T21:12:52.431238+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:52.431238+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13980940,"num_objects":1707,"num_object_clones":0,"num_object_copies":6828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":221504034,"num_objects":27044,"num_object_clones":0,"num_object_copies":108176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27044,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158034,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":443850752,"data_stored":443850752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160779,"ondisk_log_size":160779,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713660,"kb_used_data":108588,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93658180,"statfs":{"total":96636764160,"available":95905976320,"internally_reserved":0,"allocated":111194112,"data_stored":111021010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,1,2,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716620,"kb_used_data":107892,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93655220,"statfs":{"total":96636764160,"available":95902945280,"internally_reserved":0,"allocated":110481408,"data_stored":110308306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051691,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570160,"kb_used_data":109536,"kb_used_omap":4,"kb_used_meta":460603,"kb_avail":93801680,"statfs":{"total":96636764160,"available":96052920320,"internally_reserved":0,"allocated":112164864,"data_stored":111991762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":471658191},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741132,"kb_used_data":109756,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93630708,"statfs":{"total":96636764160,"available":95877844992,"internally_reserved":0,"allocated":112390144,"data_stored":112217042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,3,8,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":111931392,"data_stored":111931392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":111771648,"data_stored":111771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110497792,"data_stored":110497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:12:58.124 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10719", 'reported_seq': 10771, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.277425+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:54.277425+0000', 'last_peered': '2024-09-15T21:12:54.277425+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:54.277425+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13377536, 'num_objects': 1633, 'num_object_clones': 0, 'num_object_copies': 6532, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1633, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10719, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -643}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10697", 'reported_seq': 10783, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:55.375070+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:12:55.375070+0000', 'last_peered': '2024-09-15T21:12:55.375070+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:12:55.375070+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'600", 'ondisk_log_start': "27'600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14211742, 'num_objects': 1736, 'num_object_clones': 0, 'num_object_copies': 6944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1736, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10696, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -522}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10789", 'reported_seq': 10861, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:52.438101+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:52.438101+0000', 'last_peered': '2024-09-15T21:12:52.438101+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:52.438101+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13653260, 'num_objects': 1667, 'num_object_clones': 0, 'num_object_copies': 6668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1667, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10787, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -667}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:58.125 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'10983", 'reported_seq': 11045, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.249328+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:54.249328+0000', 'last_peered': '2024-09-15T21:12:54.249328+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:54.249328+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13726988, 'num_objects': 1676, 'num_object_clones': 0, 'num_object_copies': 6704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10981, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -635}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10740", 'reported_seq': 10790, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.127011+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:54.127011+0000', 'last_peered': '2024-09-15T21:12:54.127011+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:54.127011+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13402112, 'num_objects': 1636, 'num_object_clones': 0, 'num_object_copies': 6544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10740, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -680}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11024", 'reported_seq': 11100, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:52.539450+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:52.539450+0000', 'last_peered': '2024-09-15T21:12:52.539450+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:52.539450+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14316812, 'num_objects': 1748, 'num_object_clones': 0, 'num_object_copies': 6992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11022, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -618}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11048", 'reported_seq': 11123, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:52.587407+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:52.587407+0000', 'last_peered': '2024-09-15T21:12:52.587407+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:52.587407+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14030092, 'num_objects': 1713, 'num_object_clones': 0, 'num_object_copies': 6852, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1713, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11046, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -761}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10778", 'reported_seq': 10844, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.056377+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:54.056377+0000', 'last_peered': '2024-09-15T21:12:54.056377+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:54.056377+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13707800, 'num_objects': 1674, 'num_object_clones': 0, 'num_object_copies': 6696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1674, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10775, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -658}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.126 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10851", 'reported_seq': 10936, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:55.196197+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:12:55.196197+0000', 'last_peered': '2024-09-15T21:12:55.196197+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:12:55.196197+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14013708, 'num_objects': 1711, 'num_object_clones': 0, 'num_object_copies': 6844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1711, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10849, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -624}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10705", 'reported_seq': 10751, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.347218+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:54.347218+0000', 'last_peered': '2024-09-15T21:12:54.347218+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:54.347218+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14105222, 'num_objects': 1722, 'num_object_clones': 0, 'num_object_copies': 6888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10704, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10820", 'reported_seq': 13755, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:55.468418+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:12:55.468418+0000', 'last_peered': '2024-09-15T21:12:55.468418+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:12:55.468418+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13769350, 'num_objects': 1681, 'num_object_clones': 0, 'num_object_copies': 6724, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 863, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1681, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10819, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1425, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 818}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10800", 'reported_seq': 10863, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:54.413358+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:54.413358+0000', 'last_peered': '2024-09-15T21:12:54.413358+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:54.413358+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13366540, 'num_objects': 1632, 'num_object_clones': 0, 'num_object_copies': 6528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10798, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -647}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10914", 'reported_seq': 11003, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:55.456883+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:12:55.456883+0000', 'last_peered': '2024-09-15T21:12:55.456883+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:12:55.456883+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14472460, 'num_objects': 1767, 'num_object_clones': 0, 'num_object_copies': 7068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10912, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -514}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:12:58.127 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10842", 'reported_seq': 10916, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:52.431238+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:52.431238+0000', 'last_peered': '2024-09-15T21:12:52.431238+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:52.431238+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13980940, 'num_objects': 1707, 'num_object_clones': 0, 'num_object_copies': 6828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1707, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10840, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:12:58.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:12:58.861 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:58.861 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31017/106920 objects degraded (29.010%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":644,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":26730,"data_bytes":218931746,"bytes_used":2803666944,"bytes_avail":383743389696,"bytes_total":386547056640,"degraded_objects":31017,"degraded_total":106920,"degraded_ratio":0.29009539842873178,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:12:58.885 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:12:58.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:12:59.489 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:12:59.490 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:12:59.516 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-09-15T21:12:58.155219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":218931746,"num_objects":26730,"num_object_clones":0,"num_object_copies":106920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158348,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737956,"kb_used_data":427996,"kb_used_omap":14,"kb_used_meta":2309873,"kb_avail":374749404,"statfs":{"total":386547056640,"available":383743389696,"internally_reserved":0,"allocated":438267904,"data_stored":437575496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2365310514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,1,2,8,13,17,7,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18677760,"num_objects":-2280,"num_object_clones":0,"num_object_copies":-9120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2280,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004451"},"pg_stats":[{"pgid":"1.f","version":"35'10719","reported_seq":10771,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.277425+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:54.277425+0000","last_peered":"2024-09-15T21:12:54.277425+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:54.277425+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13377536,"num_objects":1633,"num_object_clones":0,"num_object_copies":6532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10719,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-643}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10910","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:57.980064+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:57.980064+0000","last_peered":"2024-09-15T21:12:57.980064+0000","last_clean":"2024-09-15T21:12:57.980064+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:57.980064+0000","last_undegraded":"2024-09-15T21:12:57.980064+0000","last_fullsized":"2024-09-15T21:12:57.980064+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10697","reported_seq":10783,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.375070+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:12:55.375070+0000","last_peered":"2024-09-15T21:12:55.375070+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:12:55.375070+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'600","ondisk_log_start":"27'600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14211742,"num_objects":1736,"num_object_clones":0,"num_object_copies":6944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1736,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10696,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-522}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10848","reported_seq":10922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.530349+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:57.530349+0000","last_peered":"2024-09-15T21:12:57.530349+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:57.530349+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13169932,"num_objects":1608,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'10983","reported_seq":11045,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.249328+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:54.249328+0000","last_peered":"2024-09-15T21:12:54.249328+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:54.249328+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10981,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-635}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10740","reported_seq":10790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.127011+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:54.127011+0000","last_peered":"2024-09-15T21:12:54.127011+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:54.127011+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13402112,"num_objects":1636,"num_object_clones":0,"num_object_copies":6544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10740,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-680}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11096","reported_seq":11172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.449909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:57.449909+0000","last_peered":"2024-09-15T21:12:57.449909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:57.449909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11114","reported_seq":11187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.584300+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:57.584300+0000","last_peered":"2024-09-15T21:12:57.584300+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:57.584300+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13489420,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10778","reported_seq":10844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.056377+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:54.056377+0000","last_peered":"2024-09-15T21:12:54.056377+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:54.056377+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13707800,"num_objects":1674,"num_object_clones":0,"num_object_copies":6696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10775,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-658}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10851","reported_seq":10936,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.196197+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:12:55.196197+0000","last_peered":"2024-09-15T21:12:55.196197+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:12:55.196197+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14013708,"num_objects":1711,"num_object_clones":0,"num_object_copies":6844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10849,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-624}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10713","reported_seq":15382,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:54.349183+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:54.349183+0000","last_peered":"2024-09-15T21:12:54.349183+0000","last_clean":"2024-09-15T21:12:54.349183+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:54.349183+0000","last_undegraded":"2024-09-15T21:12:54.349183+0000","last_fullsized":"2024-09-15T21:12:54.349183+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13492224,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4824,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10705","reported_seq":10751,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.347218+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:54.347218+0000","last_peered":"2024-09-15T21:12:54.347218+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:54.347218+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14105222,"num_objects":1722,"num_object_clones":0,"num_object_copies":6888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10820","reported_seq":13755,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.468418+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:12:55.468418+0000","last_peered":"2024-09-15T21:12:55.468418+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:12:55.468418+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769350,"num_objects":1681,"num_object_clones":0,"num_object_copies":6724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10819,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1425,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":818}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10800","reported_seq":10863,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:54.413358+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:54.413358+0000","last_peered":"2024-09-15T21:12:54.413358+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:54.413358+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13366540,"num_objects":1632,"num_object_clones":0,"num_object_copies":6528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10798,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-647}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10914","reported_seq":11003,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:55.456883+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:12:55.456883+0000","last_peered":"2024-09-15T21:12:55.456883+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:12:55.456883+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14472460,"num_objects":1767,"num_object_clones":0,"num_object_copies":7068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10912,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-514}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10905","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.632066+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:57.632066+0000","last_peered":"2024-09-15T21:12:57.632066+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:57.632066+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13464844,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":218931746,"num_objects":26730,"num_object_clones":0,"num_object_copies":106920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31017,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158348,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1446,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":435699712,"data_stored":435699712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160693,"ondisk_log_size":160693,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713660,"kb_used_data":108588,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93658180,"statfs":{"total":96636764160,"available":95905976320,"internally_reserved":0,"allocated":111194112,"data_stored":111021010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,1,2,1,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716620,"kb_used_data":107892,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":93655220,"statfs":{"total":96636764160,"available":95902945280,"internally_reserved":0,"allocated":110481408,"data_stored":110308306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,6,7,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570516,"kb_used_data":105732,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93801324,"statfs":{"total":96636764160,"available":96052555776,"internally_reserved":0,"allocated":108269568,"data_stored":108096466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737160,"kb_used_data":105784,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93634680,"statfs":{"total":96636764160,"available":95881912320,"internally_reserved":0,"allocated":108322816,"data_stored":108149714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107880448,"data_stored":107880448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107671552,"data_stored":107671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109649920,"data_stored":109649920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":110497792,"data_stored":110497792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:02.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:03.134 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:03.135 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:03.163 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-15T21:13:02.156947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213434914,"num_objects":26059,"num_object_clones":0,"num_object_copies":104236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159019,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738468,"kb_used_data":420184,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374748892,"statfs":{"total":386547056640,"available":383742865408,"internally_reserved":0,"allocated":430268416,"data_stored":429576008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2373830194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,8,11,13,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17924096,"num_objects":-2188,"num_object_clones":0,"num_object_copies":-8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004872"},"pg_stats":[{"pgid":"1.f","version":"35'10788","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.520355+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:59.520355+0000","last_peered":"2024-09-15T21:12:59.520355+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:59.520355+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812288,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10788,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10910","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:57.980064+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:57.980064+0000","last_peered":"2024-09-15T21:12:57.980064+0000","last_clean":"2024-09-15T21:12:57.980064+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:57.980064+0000","last_undegraded":"2024-09-15T21:12:57.980064+0000","last_fullsized":"2024-09-15T21:12:57.980064+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10751","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.454436+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:00.454436+0000","last_peered":"2024-09-15T21:13:00.454436+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:00.454436+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769374,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10750,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10848","reported_seq":10922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.530349+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:57.530349+0000","last_peered":"2024-09-15T21:12:57.530349+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:57.530349+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13169932,"num_objects":1608,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11046","reported_seq":11108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.407437+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:59.407437+0000","last_peered":"2024-09-15T21:12:59.407437+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:59.407437+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13210892,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10792","reported_seq":10843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.458765+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:59.458765+0000","last_peered":"2024-09-15T21:12:59.458765+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:59.458765+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11096","reported_seq":11172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.449909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:57.449909+0000","last_peered":"2024-09-15T21:12:57.449909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:57.449909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11114","reported_seq":11187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.584300+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:57.584300+0000","last_peered":"2024-09-15T21:12:57.584300+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:57.584300+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13489420,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10846","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.448898+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:59.448898+0000","last_peered":"2024-09-15T21:12:59.448898+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:59.448898+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150744,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10913","reported_seq":11000,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.278454+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:00.278454+0000","last_peered":"2024-09-15T21:13:00.278454+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:00.278454+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505804,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-686}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10778","reported_seq":15444,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:59.396368+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:59.396368+0000","last_peered":"2024-09-15T21:12:59.396368+0000","last_clean":"2024-09-15T21:12:59.396368+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:59.396368+0000","last_undegraded":"2024-09-15T21:12:59.396368+0000","last_fullsized":"2024-09-15T21:12:59.396368+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959744,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10772","reported_seq":10816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.305047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:59.305047+0000","last_peered":"2024-09-15T21:12:59.305047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:59.305047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556358,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10879","reported_seq":13944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.499707+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:00.499707+0000","last_peered":"2024-09-15T21:13:00.499707+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:00.499707+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13286022,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10849","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.192029+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:59.192029+0000","last_peered":"2024-09-15T21:12:59.192029+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:59.192029+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12965132,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10977","reported_seq":11066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.456016+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:00.456016+0000","last_peered":"2024-09-15T21:13:00.456016+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:00.456016+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13956364,"num_objects":1704,"num_object_clones":0,"num_object_copies":6816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10905","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.632066+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:57.632066+0000","last_peered":"2024-09-15T21:12:57.632066+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:57.632066+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13464844,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":213434914,"num_objects":26059,"num_object_clones":0,"num_object_copies":104236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159019,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":427728896,"data_stored":427728896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713896,"kb_used_data":104660,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93657944,"statfs":{"total":96636764160,"available":95905734656,"internally_reserved":0,"allocated":107171840,"data_stored":106998738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716896,"kb_used_data":104008,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93654944,"statfs":{"total":96636764160,"available":95902662656,"internally_reserved":0,"allocated":106504192,"data_stored":106331090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570516,"kb_used_data":105732,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93801324,"statfs":{"total":96636764160,"available":96052555776,"internally_reserved":0,"allocated":108269568,"data_stored":108096466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737160,"kb_used_data":105784,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93634680,"statfs":{"total":96636764160,"available":95881912320,"internally_reserved":0,"allocated":108322816,"data_stored":108149714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107880448,"data_stored":107880448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107671552,"data_stored":107671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105639936,"data_stored":105639936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106536960,"data_stored":106536960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:03.164 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:03.721 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:03.722 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:03.754 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-15T21:13:02.156947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":213434914,"num_objects":26059,"num_object_clones":0,"num_object_copies":104236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159019,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738468,"kb_used_data":420184,"kb_used_omap":14,"kb_used_meta":2318193,"kb_avail":374748892,"statfs":{"total":386547056640,"available":383742865408,"internally_reserved":0,"allocated":430268416,"data_stored":429576008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2373830194},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,3,8,11,13,16,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17924096,"num_objects":-2188,"num_object_clones":0,"num_object_copies":-8752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004872"},"pg_stats":[{"pgid":"1.f","version":"35'10788","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.520355+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:59.520355+0000","last_peered":"2024-09-15T21:12:59.520355+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:59.520355+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812288,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10788,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10910","reported_seq":15801,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:57.980064+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:12:57.980064+0000","last_peered":"2024-09-15T21:12:57.980064+0000","last_clean":"2024-09-15T21:12:57.980064+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:12:57.980064+0000","last_undegraded":"2024-09-15T21:12:57.980064+0000","last_fullsized":"2024-09-15T21:12:57.980064+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13434880,"num_objects":1640,"num_object_clones":0,"num_object_copies":6560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1576,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10751","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.454436+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:00.454436+0000","last_peered":"2024-09-15T21:13:00.454436+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:00.454436+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769374,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10750,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10848","reported_seq":10922,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.530349+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:12:57.530349+0000","last_peered":"2024-09-15T21:12:57.530349+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:12:57.530349+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13169932,"num_objects":1608,"num_object_clones":0,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10846,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11046","reported_seq":11108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.407437+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:59.407437+0000","last_peered":"2024-09-15T21:12:59.407437+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:59.407437+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13210892,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10792","reported_seq":10843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.458765+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:59.458765+0000","last_peered":"2024-09-15T21:12:59.458765+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:59.458765+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11096","reported_seq":11172,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.449909+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:12:57.449909+0000","last_peered":"2024-09-15T21:12:57.449909+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:12:57.449909+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13726988,"num_objects":1676,"num_object_clones":0,"num_object_copies":6704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-690}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11114","reported_seq":11187,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.584300+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:12:57.584300+0000","last_peered":"2024-09-15T21:12:57.584300+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:12:57.584300+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13489420,"num_objects":1647,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11112,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10846","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.448898+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:59.448898+0000","last_peered":"2024-09-15T21:12:59.448898+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:59.448898+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150744,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10913","reported_seq":11000,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.278454+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:00.278454+0000","last_peered":"2024-09-15T21:13:00.278454+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:00.278454+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505804,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-686}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10778","reported_seq":15444,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:59.396368+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:59.396368+0000","last_peered":"2024-09-15T21:12:59.396368+0000","last_clean":"2024-09-15T21:12:59.396368+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:59.396368+0000","last_undegraded":"2024-09-15T21:12:59.396368+0000","last_fullsized":"2024-09-15T21:12:59.396368+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959744,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10772","reported_seq":10816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.305047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:59.305047+0000","last_peered":"2024-09-15T21:12:59.305047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:59.305047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556358,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10879","reported_seq":13944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.499707+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:00.499707+0000","last_peered":"2024-09-15T21:13:00.499707+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:00.499707+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13286022,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10849","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.192029+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:59.192029+0000","last_peered":"2024-09-15T21:12:59.192029+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:59.192029+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12965132,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10977","reported_seq":11066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.456016+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:00.456016+0000","last_peered":"2024-09-15T21:13:00.456016+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:00.456016+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13956364,"num_objects":1704,"num_object_clones":0,"num_object_copies":6816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10905","reported_seq":10980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:57.632066+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:12:57.632066+0000","last_peered":"2024-09-15T21:12:57.632066+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:12:57.632066+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13464844,"num_objects":1644,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10903,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-671}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":213434914,"num_objects":26059,"num_object_clones":0,"num_object_copies":104236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26059,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159019,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":427728896,"data_stored":427728896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160864,"ondisk_log_size":160864,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713896,"kb_used_data":104660,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93657944,"statfs":{"total":96636764160,"available":95905734656,"internally_reserved":0,"allocated":107171840,"data_stored":106998738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716896,"kb_used_data":104008,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93654944,"statfs":{"total":96636764160,"available":95902662656,"internally_reserved":0,"allocated":106504192,"data_stored":106331090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051692,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570516,"kb_used_data":105732,"kb_used_omap":4,"kb_used_meta":464763,"kb_avail":93801324,"statfs":{"total":96636764160,"available":96052555776,"internally_reserved":0,"allocated":108269568,"data_stored":108096466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":475918031},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,5,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737160,"kb_used_data":105784,"kb_used_omap":4,"kb_used_meta":631355,"kb_avail":93634680,"statfs":{"total":96636764160,"available":95881912320,"internally_reserved":0,"allocated":108322816,"data_stored":108149714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":646508238},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":107880448,"data_stored":107880448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107671552,"data_stored":107671552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105639936,"data_stored":105639936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106536960,"data_stored":106536960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:03.755 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10788", 'reported_seq': 10837, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.520355+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:12:59.520355+0000', 'last_peered': '2024-09-15T21:12:59.520355+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:12:59.520355+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12812288, 'num_objects': 1564, 'num_object_clones': 0, 'num_object_copies': 6256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1564, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10788, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -712}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10751", 'reported_seq': 10837, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:00.454436+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:00.454436+0000', 'last_peered': '2024-09-15T21:13:00.454436+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:00.454436+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13769374, 'num_objects': 1682, 'num_object_clones': 0, 'num_object_copies': 6728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1682, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10750, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -576}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10848", 'reported_seq': 10922, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:57.530349+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:12:57.530349+0000', 'last_peered': '2024-09-15T21:12:57.530349+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:12:57.530349+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13169932, 'num_objects': 1608, 'num_object_clones': 0, 'num_object_copies': 6432, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1608, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10846, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11046", 'reported_seq': 11108, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.407437+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:12:59.407437+0000', 'last_peered': '2024-09-15T21:12:59.407437+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:12:59.407437+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13210892, 'num_objects': 1613, 'num_object_clones': 0, 'num_object_copies': 6452, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1613, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11044, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -698}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:03.756 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10792", 'reported_seq': 10843, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.458765+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:12:59.458765+0000', 'last_peered': '2024-09-15T21:12:59.458765+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:12:59.458765+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12976128, 'num_objects': 1584, 'num_object_clones': 0, 'num_object_copies': 6336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1584, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10792, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -732}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11096", 'reported_seq': 11172, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:57.449909+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:12:57.449909+0000', 'last_peered': '2024-09-15T21:12:57.449909+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:12:57.449909+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13726988, 'num_objects': 1676, 'num_object_clones': 0, 'num_object_copies': 6704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1676, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11094, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -690}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11114", 'reported_seq': 11187, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:57.584300+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:12:57.584300+0000', 'last_peered': '2024-09-15T21:12:57.584300+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:12:57.584300+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13489420, 'num_objects': 1647, 'num_object_clones': 0, 'num_object_copies': 6588, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1647, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11112, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -827}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10846", 'reported_seq': 10912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.448898+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:12:59.448898+0000', 'last_peered': '2024-09-15T21:12:59.448898+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:12:59.448898+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13150744, 'num_objects': 1606, 'num_object_clones': 0, 'num_object_copies': 6424, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1606, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10843, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -726}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10913", 'reported_seq': 11000, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:00.278454+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:00.278454+0000', 'last_peered': '2024-09-15T21:13:00.278454+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:00.278454+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13505804, 'num_objects': 1649, 'num_object_clones': 0, 'num_object_copies': 6596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1649, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10911, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -686}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:03.757 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10772", 'reported_seq': 10816, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.305047+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:12:59.305047+0000', 'last_peered': '2024-09-15T21:12:59.305047+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:12:59.305047+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'700", 'ondisk_log_start': "27'700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13556358, 'num_objects': 1655, 'num_object_clones': 0, 'num_object_copies': 6620, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1655, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10771, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -601}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10879", 'reported_seq': 13944, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:00.499707+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:00.499707+0000', 'last_peered': '2024-09-15T21:13:00.499707+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:00.499707+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13286022, 'num_objects': 1622, 'num_object_clones': 0, 'num_object_copies': 6488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 797, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1622, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10878, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1491, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 825}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10849", 'reported_seq': 10912, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:59.192029+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:12:59.192029+0000', 'last_peered': '2024-09-15T21:12:59.192029+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:12:59.192029+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12965132, 'num_objects': 1583, 'num_object_clones': 0, 'num_object_copies': 6332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1583, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10847, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -696}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'10977", 'reported_seq': 11066, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:00.456016+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:00.456016+0000', 'last_peered': '2024-09-15T21:13:00.456016+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:00.456016+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13956364, 'num_objects': 1704, 'num_object_clones': 0, 'num_object_copies': 6816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10975, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:03.758 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'10905", 'reported_seq': 10980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:12:57.632066+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:12:57.632066+0000', 'last_peered': '2024-09-15T21:12:57.632066+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:12:57.632066+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13464844, 'num_objects': 1644, 'num_object_clones': 0, 'num_object_copies': 6576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10903, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -671}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:03.759 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:04.490 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:04.490 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30951/103104 objects degraded (30.019%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":650,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":25776,"data_bytes":211116578,"bytes_used":2804637696,"bytes_avail":383742418944,"bytes_total":386547056640,"degraded_objects":30951,"degraded_total":103104,"degraded_ratio":0.30019203910614523,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":185},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:04.517 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:04.518 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:05.127 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:05.128 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:05.169 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-15T21:13:04.157755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":211116578,"num_objects":25776,"num_object_clones":0,"num_object_copies":103104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159302,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160947,"ondisk_log_size":160947,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738904,"kb_used_data":412292,"kb_used_omap":14,"kb_used_meta":2326513,"kb_avail":374748456,"statfs":{"total":386547056640,"available":383742418944,"internally_reserved":0,"allocated":422187008,"data_stored":421494600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2382349874},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,12,15,13,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18284544,"num_objects":-2232,"num_object_clones":0,"num_object_copies":-8928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004795"},"pg_stats":[{"pgid":"1.f","version":"35'10788","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.520355+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:12:59.520355+0000","last_peered":"2024-09-15T21:12:59.520355+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:12:59.520355+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12812288,"num_objects":1564,"num_object_clones":0,"num_object_copies":6256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10788,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-712}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10973","reported_seq":15866,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:02.956713+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:02.956713+0000","last_peered":"2024-09-15T21:13:02.956713+0000","last_clean":"2024-09-15T21:13:02.956713+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:02.956713+0000","last_undegraded":"2024-09-15T21:13:02.956713+0000","last_fullsized":"2024-09-15T21:13:02.956713+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12918784,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10751","reported_seq":10837,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.454436+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:00.454436+0000","last_peered":"2024-09-15T21:13:00.454436+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:00.454436+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13769374,"num_objects":1682,"num_object_clones":0,"num_object_copies":6728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10750,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-576}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10903","reported_seq":10977,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:02.666393+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:02.666393+0000","last_peered":"2024-09-15T21:13:02.666393+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:02.666393+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12719372,"num_objects":1553,"num_object_clones":0,"num_object_copies":6212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10901,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-781}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11046","reported_seq":11108,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.407437+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:12:59.407437+0000","last_peered":"2024-09-15T21:12:59.407437+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:12:59.407437+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13210892,"num_objects":1613,"num_object_clones":0,"num_object_copies":6452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-698}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10792","reported_seq":10843,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.458765+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:12:59.458765+0000","last_peered":"2024-09-15T21:12:59.458765+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:12:59.458765+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12976128,"num_objects":1584,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10792,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-732}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11158","reported_seq":11234,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:02.602290+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:02.602290+0000","last_peered":"2024-09-15T21:13:02.602290+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:02.602290+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13219084,"num_objects":1614,"num_object_clones":0,"num_object_copies":6456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-752}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11165","reported_seq":11240,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:02.412347+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:02.412347+0000","last_peered":"2024-09-15T21:13:02.412347+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:02.412347+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13071628,"num_objects":1596,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11163,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-878}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10846","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.448898+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:12:59.448898+0000","last_peered":"2024-09-15T21:12:59.448898+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:12:59.448898+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13150744,"num_objects":1606,"num_object_clones":0,"num_object_copies":6424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10843,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-726}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10913","reported_seq":11000,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.278454+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:00.278454+0000","last_peered":"2024-09-15T21:13:00.278454+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:00.278454+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13505804,"num_objects":1649,"num_object_clones":0,"num_object_copies":6596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10911,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-686}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10778","reported_seq":15444,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:12:59.396368+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:12:59.396368+0000","last_peered":"2024-09-15T21:12:59.396368+0000","last_clean":"2024-09-15T21:12:59.396368+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:12:59.396368+0000","last_undegraded":"2024-09-15T21:12:59.396368+0000","last_fullsized":"2024-09-15T21:12:59.396368+0000","mapping_epoch":32,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12959744,"num_objects":1582,"num_object_clones":0,"num_object_copies":6328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4889,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10772","reported_seq":10816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.305047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:12:59.305047+0000","last_peered":"2024-09-15T21:12:59.305047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:12:59.305047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'700","ondisk_log_start":"27'700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13556358,"num_objects":1655,"num_object_clones":0,"num_object_copies":6620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10771,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-601}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10879","reported_seq":13944,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.499707+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:00.499707+0000","last_peered":"2024-09-15T21:13:00.499707+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:00.499707+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13286022,"num_objects":1622,"num_object_clones":0,"num_object_copies":6488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":797,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10878,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":825}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10849","reported_seq":10912,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:12:59.192029+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:12:59.192029+0000","last_peered":"2024-09-15T21:12:59.192029+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:12:59.192029+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12965132,"num_objects":1583,"num_object_clones":0,"num_object_copies":6332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10847,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-696}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'10977","reported_seq":11066,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:00.456016+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:00.456016+0000","last_peered":"2024-09-15T21:13:00.456016+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:00.456016+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13956364,"num_objects":1704,"num_object_clones":0,"num_object_copies":6816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'10957","reported_seq":11031,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:02.502647+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:02.502647+0000","last_peered":"2024-09-15T21:13:02.502647+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:02.502647+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13038860,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10955,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-723}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":211116578,"num_objects":25776,"num_object_clones":0,"num_object_copies":103104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25776,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159302,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1512,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":419676160,"data_stored":419676160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160947,"ondisk_log_size":160947,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713896,"kb_used_data":104660,"kb_used_omap":4,"kb_used_meta":609211,"kb_avail":93657944,"statfs":{"total":96636764160,"available":95905734656,"internally_reserved":0,"allocated":107171840,"data_stored":106998738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":623832784},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,4,3,8],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716896,"kb_used_data":104008,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93654944,"statfs":{"total":96636764160,"available":95902662656,"internally_reserved":0,"allocated":106504192,"data_stored":106331090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,4,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570708,"kb_used_data":101760,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93801132,"statfs":{"total":96636764160,"available":96052359168,"internally_reserved":0,"allocated":104202240,"data_stored":104029138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247931,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":737404,"kb_used_data":101864,"kb_used_omap":4,"kb_used_meta":635515,"kb_avail":93634436,"statfs":{"total":96636764160,"available":95881662464,"internally_reserved":0,"allocated":104308736,"data_stored":104135634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":650768078},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":103858176,"data_stored":103858176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103641088,"data_stored":103641088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105639936,"data_stored":105639936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106536960,"data_stored":106536960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:08.171 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:08.782 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:08.782 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:08.806 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-15T21:13:08.159361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203383330,"num_objects":24832,"num_object_clones":0,"num_object_copies":99328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160246,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722996,"kb_used_data":400556,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374764364,"statfs":{"total":386547056640,"available":383758708736,"internally_reserved":0,"allocated":410169344,"data_stored":409476936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2378090034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,2,4,15,16,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18120704,"num_objects":-2212,"num_object_clones":0,"num_object_copies":-8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004835"},"pg_stats":[{"pgid":"1.f","version":"35'10852","reported_seq":10904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.424608+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:04.424608+0000","last_peered":"2024-09-15T21:13:04.424608+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:04.424608+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12288000,"num_objects":1500,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10973","reported_seq":15866,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:02.956713+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:02.956713+0000","last_peered":"2024-09-15T21:13:02.956713+0000","last_clean":"2024-09-15T21:13:02.956713+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:02.956713+0000","last_undegraded":"2024-09-15T21:13:02.956713+0000","last_fullsized":"2024-09-15T21:13:02.956713+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12918784,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10812","reported_seq":10898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.317908+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:05.317908+0000","last_peered":"2024-09-15T21:13:05.317908+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:05.317908+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13269662,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10811,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10965","reported_seq":11040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.665412+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:07.665412+0000","last_peered":"2024-09-15T21:13:07.665412+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:07.665412+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12211468,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11100","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.387998+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:04.387998+0000","last_peered":"2024-09-15T21:13:04.387998+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:04.387998+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12768524,"num_objects":1559,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10858","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.468993+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:04.468993+0000","last_peered":"2024-09-15T21:13:04.468993+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:04.468993+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12435456,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11231","reported_seq":11307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.526771+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:07.526771+0000","last_peered":"2024-09-15T21:13:07.526771+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:07.526771+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12621068,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11221","reported_seq":11296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.511972+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:07.511972+0000","last_peered":"2024-09-15T21:13:07.511972+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:07.511972+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10913","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.396828+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:04.396828+0000","last_peered":"2024-09-15T21:13:04.396828+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:04.396828+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12601880,"num_objects":1539,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10910,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10973","reported_seq":11060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.474165+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:05.474165+0000","last_peered":"2024-09-15T21:13:05.474165+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:05.474165+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10834","reported_seq":15503,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:04.460212+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:04.460212+0000","last_peered":"2024-09-15T21:13:04.460212+0000","last_clean":"2024-09-15T21:13:04.460212+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:04.460212+0000","last_undegraded":"2024-09-15T21:13:04.460212+0000","last_fullsized":"2024-09-15T21:13:04.460212+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12500992,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10835","reported_seq":10881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.391376+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:04.391376+0000","last_peered":"2024-09-15T21:13:04.391376+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:04.391376+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13040262,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10940","reported_seq":14123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.571583+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:05.571583+0000","last_peered":"2024-09-15T21:13:05.571583+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:05.571583+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12786310,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10907","reported_seq":10970,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.234904+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:04.234904+0000","last_peered":"2024-09-15T21:13:04.234904+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:04.234904+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489996,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-754}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11052","reported_seq":11141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.446052+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:05.446052+0000","last_peered":"2024-09-15T21:13:05.446052+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:05.446052+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13341964,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11025","reported_seq":11096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.690343+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:07.690343+0000","last_peered":"2024-09-15T21:13:07.690343+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:07.690343+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12481804,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":203383330,"num_objects":24832,"num_object_clones":0,"num_object_copies":99328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160246,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407764992,"data_stored":407764992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714028,"kb_used_data":100636,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93657812,"statfs":{"total":96636764160,"available":95905599488,"internally_reserved":0,"allocated":103051264,"data_stored":102878162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,0,2,3,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717188,"kb_used_data":100148,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93654652,"statfs":{"total":96636764160,"available":95902363648,"internally_reserved":0,"allocated":102551552,"data_stored":102378450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570708,"kb_used_data":101760,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93801132,"statfs":{"total":96636764160,"available":96052359168,"internally_reserved":0,"allocated":104202240,"data_stored":104029138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721072,"kb_used_data":98012,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93650768,"statfs":{"total":96636764160,"available":95898386432,"internally_reserved":0,"allocated":100364288,"data_stored":100191186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":637988558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99917824,"data_stored":99917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103641088,"data_stored":103641088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101683200,"data_stored":101683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102522880,"data_stored":102522880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:08.808 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:09.404 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:09.405 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:09.442 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-15T21:13:08.159361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203383330,"num_objects":24832,"num_object_clones":0,"num_object_copies":99328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160246,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722996,"kb_used_data":400556,"kb_used_omap":14,"kb_used_meta":2322353,"kb_avail":374764364,"statfs":{"total":386547056640,"available":383758708736,"internally_reserved":0,"allocated":410169344,"data_stored":409476936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2378090034},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,4,2,4,15,16,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18120704,"num_objects":-2212,"num_object_clones":0,"num_object_copies":-8848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004835"},"pg_stats":[{"pgid":"1.f","version":"35'10852","reported_seq":10904,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.424608+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:04.424608+0000","last_peered":"2024-09-15T21:13:04.424608+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:04.424608+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12288000,"num_objects":1500,"num_object_clones":0,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10852,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-776}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'10973","reported_seq":15866,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:02.956713+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:02.956713+0000","last_peered":"2024-09-15T21:13:02.956713+0000","last_clean":"2024-09-15T21:13:02.956713+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:02.956713+0000","last_undegraded":"2024-09-15T21:13:02.956713+0000","last_fullsized":"2024-09-15T21:13:02.956713+0000","mapping_epoch":34,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12918784,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1639,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10812","reported_seq":10898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.317908+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:05.317908+0000","last_peered":"2024-09-15T21:13:05.317908+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:05.317908+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13269662,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10811,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10965","reported_seq":11040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.665412+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:07.665412+0000","last_peered":"2024-09-15T21:13:07.665412+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:07.665412+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12211468,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11100","reported_seq":11162,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.387998+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:04.387998+0000","last_peered":"2024-09-15T21:13:04.387998+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:04.387998+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12768524,"num_objects":1559,"num_object_clones":0,"num_object_copies":6236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-752}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10858","reported_seq":10909,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.468993+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:04.468993+0000","last_peered":"2024-09-15T21:13:04.468993+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:04.468993+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12435456,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10858,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-798}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11231","reported_seq":11307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.526771+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:07.526771+0000","last_peered":"2024-09-15T21:13:07.526771+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:07.526771+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12621068,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11221","reported_seq":11296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.511972+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:07.511972+0000","last_peered":"2024-09-15T21:13:07.511972+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:07.511972+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10913","reported_seq":10979,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.396828+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:04.396828+0000","last_peered":"2024-09-15T21:13:04.396828+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:04.396828+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12601880,"num_objects":1539,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10910,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-793}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10973","reported_seq":11060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.474165+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:05.474165+0000","last_peered":"2024-09-15T21:13:05.474165+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:05.474165+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10834","reported_seq":15503,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:04.460212+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:04.460212+0000","last_peered":"2024-09-15T21:13:04.460212+0000","last_clean":"2024-09-15T21:13:04.460212+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:04.460212+0000","last_undegraded":"2024-09-15T21:13:04.460212+0000","last_fullsized":"2024-09-15T21:13:04.460212+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12500992,"num_objects":1526,"num_object_clones":0,"num_object_copies":6104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4945,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10835","reported_seq":10881,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.391376+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:04.391376+0000","last_peered":"2024-09-15T21:13:04.391376+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:04.391376+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13040262,"num_objects":1592,"num_object_clones":0,"num_object_copies":6368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10834,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-664}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10940","reported_seq":14123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.571583+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:05.571583+0000","last_peered":"2024-09-15T21:13:05.571583+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:05.571583+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12786310,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10907","reported_seq":10970,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:04.234904+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:04.234904+0000","last_peered":"2024-09-15T21:13:04.234904+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:04.234904+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12489996,"num_objects":1525,"num_object_clones":0,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-754}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11052","reported_seq":11141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.446052+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:05.446052+0000","last_peered":"2024-09-15T21:13:05.446052+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:05.446052+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13341964,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11025","reported_seq":11096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.690343+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:07.690343+0000","last_peered":"2024-09-15T21:13:07.690343+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:07.690343+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12481804,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":203383330,"num_objects":24832,"num_object_clones":0,"num_object_copies":99328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160246,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":407764992,"data_stored":407764992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160691,"ondisk_log_size":160691,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714028,"kb_used_data":100636,"kb_used_omap":4,"kb_used_meta":613371,"kb_avail":93657812,"statfs":{"total":96636764160,"available":95905599488,"internally_reserved":0,"allocated":103051264,"data_stored":102878162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":628092624},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,0,2,3,4,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717188,"kb_used_data":100148,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93654652,"statfs":{"total":96636764160,"available":95902363648,"internally_reserved":0,"allocated":102551552,"data_stored":102378450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051693,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":570708,"kb_used_data":101760,"kb_used_omap":4,"kb_used_meta":468923,"kb_avail":93801132,"statfs":{"total":96636764160,"available":96052359168,"internally_reserved":0,"allocated":104202240,"data_stored":104029138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":480177871},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,7,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721072,"kb_used_data":98012,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93650768,"statfs":{"total":96636764160,"available":95898386432,"internally_reserved":0,"allocated":100364288,"data_stored":100191186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":637988558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99917824,"data_stored":99917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":103641088,"data_stored":103641088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101683200,"data_stored":101683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":102522880,"data_stored":102522880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10852", 'reported_seq': 10904, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.424608+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:04.424608+0000', 'last_peered': '2024-09-15T21:13:04.424608+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:04.424608+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12288000, 'num_objects': 1500, 'num_object_clones': 0, 'num_object_copies': 6000, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1500, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10852, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -776}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10812", 'reported_seq': 10898, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:05.317908+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:05.317908+0000', 'last_peered': '2024-09-15T21:13:05.317908+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:05.317908+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10012, 'log_dups_size': 0, 'ondisk_log_size': 10012, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13269662, 'num_objects': 1621, 'num_object_clones': 0, 'num_object_copies': 6484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1621, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10811, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -637}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'10965", 'reported_seq': 11040, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:07.665412+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:07.665412+0000', 'last_peered': '2024-09-15T21:13:07.665412+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:07.665412+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12211468, 'num_objects': 1491, 'num_object_clones': 0, 'num_object_copies': 5964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1491, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10963, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:09.443 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11100", 'reported_seq': 11162, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.387998+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:04.387998+0000', 'last_peered': '2024-09-15T21:13:04.387998+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:04.387998+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12768524, 'num_objects': 1559, 'num_object_clones': 0, 'num_object_copies': 6236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11098, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -752}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10858", 'reported_seq': 10909, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.468993+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:04.468993+0000', 'last_peered': '2024-09-15T21:13:04.468993+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:04.468993+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12435456, 'num_objects': 1518, 'num_object_clones': 0, 'num_object_copies': 6072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10858, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -798}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11231", 'reported_seq': 11307, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:07.526771+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:07.526771+0000', 'last_peered': '2024-09-15T21:13:07.526771+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:07.526771+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12621068, 'num_objects': 1541, 'num_object_clones': 0, 'num_object_copies': 6164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1541, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11229, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -825}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11221", 'reported_seq': 11296, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:07.511972+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:07.511972+0000', 'last_peered': '2024-09-15T21:13:07.511972+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:07.511972+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12612876, 'num_objects': 1540, 'num_object_clones': 0, 'num_object_copies': 6160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1540, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11219, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10913", 'reported_seq': 10979, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.396828+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:04.396828+0000', 'last_peered': '2024-09-15T21:13:04.396828+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:04.396828+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12601880, 'num_objects': 1539, 'num_object_clones': 0, 'num_object_copies': 6156, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1539, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10910, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -793}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:09.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'10973", 'reported_seq': 11060, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:05.474165+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:05.474165+0000', 'last_peered': '2024-09-15T21:13:05.474165+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:05.474165+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13014284, 'num_objects': 1589, 'num_object_clones': 0, 'num_object_copies': 6356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10971, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -746}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10835", 'reported_seq': 10881, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.391376+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:04.391376+0000', 'last_peered': '2024-09-15T21:13:04.391376+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:04.391376+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13040262, 'num_objects': 1592, 'num_object_clones': 0, 'num_object_copies': 6368, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1592, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10834, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -664}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'10940", 'reported_seq': 14123, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:05.571583+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:05.571583+0000', 'last_peered': '2024-09-15T21:13:05.571583+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:05.571583+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12786310, 'num_objects': 1561, 'num_object_clones': 0, 'num_object_copies': 6244, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 739, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1561, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10939, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1549, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 822}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10907", 'reported_seq': 10970, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:04.234904+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:04.234904+0000', 'last_peered': '2024-09-15T21:13:04.234904+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:04.234904+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12489996, 'num_objects': 1525, 'num_object_clones': 0, 'num_object_copies': 6100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10905, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11052", 'reported_seq': 11141, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:05.446052+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:05.446052+0000', 'last_peered': '2024-09-15T21:13:05.446052+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:05.446052+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13341964, 'num_objects': 1629, 'num_object_clones': 0, 'num_object_copies': 6516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1629, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11050, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -652}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:09.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11025", 'reported_seq': 11096, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:07.690343+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:07.690343+0000', 'last_peered': '2024-09-15T21:13:07.690343+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:07.690343+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12481804, 'num_objects': 1524, 'num_object_clones': 0, 'num_object_copies': 6096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1524, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11023, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -791}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:09.446 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:10.198 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:10.199 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30893/99328 objects degraded (31.102%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":655,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":24832,"data_bytes":203383330,"bytes_used":2788347904,"bytes_avail":383758708736,"bytes_total":386547056640,"degraded_objects":30893,"degraded_total":99328,"degraded_ratio":0.31102005476804123,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:10.221 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:10.221 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:10.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:10.804 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:10.821 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-15T21:13:10.160047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":199336482,"num_objects":24338,"num_object_clones":0,"num_object_copies":97352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160740,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2741748,"kb_used_data":392924,"kb_used_omap":14,"kb_used_meta":2348721,"kb_avail":374745612,"statfs":{"total":386547056640,"available":383739506688,"internally_reserved":0,"allocated":402354176,"data_stored":401661768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2405090866},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,4,7,6,16,11,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19595264,"num_objects":-2392,"num_object_clones":0,"num_object_copies":-9568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2392,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004827"},"pg_stats":[{"pgid":"1.f","version":"35'10915","reported_seq":10967,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.620437+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:09.620437+0000","last_peered":"2024-09-15T21:13:09.620437+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:09.620437+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11771904,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11033","reported_seq":15925,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:08.121142+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:08.121142+0000","last_peered":"2024-09-15T21:13:08.121142+0000","last_clean":"2024-09-15T21:13:08.121142+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:08.121142+0000","last_undegraded":"2024-09-15T21:13:08.121142+0000","last_fullsized":"2024-09-15T21:13:08.121142+0000","mapping_epoch":34,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12427264,"num_objects":1517,"num_object_clones":0,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1699,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10812","reported_seq":10898,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.317908+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:05.317908+0000","last_peered":"2024-09-15T21:13:05.317908+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:05.317908+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13269662,"num_objects":1621,"num_object_clones":0,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10811,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-637}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'10965","reported_seq":11040,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.665412+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:07.665412+0000","last_peered":"2024-09-15T21:13:07.665412+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:07.665412+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12211468,"num_objects":1491,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11169","reported_seq":11231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.419182+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:09.419182+0000","last_peered":"2024-09-15T21:13:09.419182+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:09.419182+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203276,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10920","reported_seq":10971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.601339+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:09.601339+0000","last_peered":"2024-09-15T21:13:09.601339+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:09.601339+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11231","reported_seq":11307,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.526771+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:07.526771+0000","last_peered":"2024-09-15T21:13:07.526771+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:07.526771+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12621068,"num_objects":1541,"num_object_clones":0,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11229,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-825}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11221","reported_seq":11296,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.511972+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:07.511972+0000","last_peered":"2024-09-15T21:13:07.511972+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:07.511972+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12612876,"num_objects":1540,"num_object_clones":0,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11219,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10969","reported_seq":11035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.479378+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:09.479378+0000","last_peered":"2024-09-15T21:13:09.479378+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:09.479378+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12143128,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-849}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'10973","reported_seq":11060,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.474165+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:05.474165+0000","last_peered":"2024-09-15T21:13:05.474165+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:05.474165+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13014284,"num_objects":1589,"num_object_clones":0,"num_object_copies":6356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10971,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-746}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10900","reported_seq":15569,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:09.536451+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:09.536451+0000","last_peered":"2024-09-15T21:13:09.536451+0000","last_clean":"2024-09-15T21:13:09.536451+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:09.536451+0000","last_undegraded":"2024-09-15T21:13:09.536451+0000","last_fullsized":"2024-09-15T21:13:09.536451+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10897","reported_seq":10943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.619054+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:09.619054+0000","last_peered":"2024-09-15T21:13:09.619054+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:09.619054+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12532358,"num_objects":1530,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'10940","reported_seq":14123,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.571583+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:05.571583+0000","last_peered":"2024-09-15T21:13:05.571583+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:05.571583+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12786310,"num_objects":1561,"num_object_clones":0,"num_object_copies":6244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":739,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10939,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":822}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10963","reported_seq":11026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.546325+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:09.546325+0000","last_peered":"2024-09-15T21:13:09.546325+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:09.546325+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12031244,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11052","reported_seq":11141,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:05.446052+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:05.446052+0000","last_peered":"2024-09-15T21:13:05.446052+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:05.446052+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13341964,"num_objects":1629,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11050,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-652}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11025","reported_seq":11096,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:07.690343+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:07.690343+0000","last_peered":"2024-09-15T21:13:07.690343+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:07.690343+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12481804,"num_objects":1524,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-791}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":199336482,"num_objects":24338,"num_object_clones":0,"num_object_copies":97352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30893,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24338,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160740,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1570,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":399585280,"data_stored":399585280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":96856,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":99180544,"data_stored":99007442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717188,"kb_used_data":100148,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93654652,"statfs":{"total":96636764160,"available":95902363648,"internally_reserved":0,"allocated":102551552,"data_stored":102378450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051694,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":589060,"kb_used_data":97908,"kb_used_omap":4,"kb_used_meta":491131,"kb_avail":93782780,"statfs":{"total":96636764160,"available":96033566720,"internally_reserved":0,"allocated":100257792,"data_stored":100084690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":502918863},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,1,6,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247932,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721072,"kb_used_data":98012,"kb_used_omap":4,"kb_used_meta":623035,"kb_avail":93650768,"statfs":{"total":96636764160,"available":95898386432,"internally_reserved":0,"allocated":100364288,"data_stored":100191186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":637988558},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,0,2,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":99917824,"data_stored":99917824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":99586048,"data_stored":99586048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101683200,"data_stored":101683200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98398208,"data_stored":98398208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:13.824 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:14.450 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:14.451 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:14.477 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T21:13:14.161702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194748962,"num_objects":23778,"num_object_clones":0,"num_object_copies":95112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161300,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709280,"kb_used_data":381252,"kb_used_omap":14,"kb_used_meta":2327921,"kb_avail":374778080,"statfs":{"total":386547056640,"available":383772753920,"internally_reserved":0,"allocated":390402048,"data_stored":389709640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2383791666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,7,13,15,14,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004755"},"pg_stats":[{"pgid":"1.f","version":"35'10915","reported_seq":10967,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.620437+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:09.620437+0000","last_peered":"2024-09-15T21:13:09.620437+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:09.620437+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11771904,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11108","reported_seq":16002,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:13.167877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:13.167877+0000","last_peered":"2024-09-15T21:13:13.167877+0000","last_clean":"2024-09-15T21:13:13.167877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:13.167877+0000","last_undegraded":"2024-09-15T21:13:13.167877+0000","last_fullsized":"2024-09-15T21:13:13.167877+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11812864,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10888","reported_seq":10974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.492361+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:10.492361+0000","last_peered":"2024-09-15T21:13:10.492361+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:10.492361+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12647070,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10887,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11022","reported_seq":11097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.776697+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:12.776697+0000","last_peered":"2024-09-15T21:13:12.776697+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:12.776697+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11744524,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11169","reported_seq":11231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.419182+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:09.419182+0000","last_peered":"2024-09-15T21:13:09.419182+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:09.419182+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203276,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10920","reported_seq":10971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.601339+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:09.601339+0000","last_peered":"2024-09-15T21:13:09.601339+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:09.601339+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11297","reported_seq":11373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.588411+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:12.588411+0000","last_peered":"2024-09-15T21:13:12.588411+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:12.588411+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080396,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-891}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11279","reported_seq":11354,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.732608+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:12.732608+0000","last_peered":"2024-09-15T21:13:12.732608+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:12.732608+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137740,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10969","reported_seq":11035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.479378+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:09.479378+0000","last_peered":"2024-09-15T21:13:09.479378+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:09.479378+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12143128,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-849}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11034","reported_seq":11120,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.583177+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:10.583177+0000","last_peered":"2024-09-15T21:13:10.583177+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:10.583177+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12514572,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11032,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-807}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10900","reported_seq":15569,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:09.536451+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:09.536451+0000","last_peered":"2024-09-15T21:13:09.536451+0000","last_clean":"2024-09-15T21:13:09.536451+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:09.536451+0000","last_undegraded":"2024-09-15T21:13:09.536451+0000","last_fullsized":"2024-09-15T21:13:09.536451+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10897","reported_seq":10943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.619054+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:09.619054+0000","last_peered":"2024-09-15T21:13:09.619054+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:09.619054+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12532358,"num_objects":1530,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11003","reported_seq":14315,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.635161+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:10.635161+0000","last_peered":"2024-09-15T21:13:10.635161+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:10.635161+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12270214,"num_objects":1498,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10963","reported_seq":11026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.546325+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:09.546325+0000","last_peered":"2024-09-15T21:13:09.546325+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:09.546325+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12031244,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11104","reported_seq":11191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.405257+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:10.405257+0000","last_peered":"2024-09-15T21:13:10.405257+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:10.405257+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12915980,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11077","reported_seq":11152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.548584+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:12.548584+0000","last_peered":"2024-09-15T21:13:12.548584+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:12.548584+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194748962,"num_objects":23778,"num_object_clones":0,"num_object_copies":95112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161300,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":387477504,"data_stored":387477504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":96856,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":99180544,"data_stored":99007442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700832,"kb_used_data":96268,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93671008,"statfs":{"total":96636764160,"available":95919112192,"internally_reserved":0,"allocated":98578432,"data_stored":98405330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572700,"kb_used_data":94028,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93799140,"statfs":{"total":96636764160,"available":96050319360,"internally_reserved":0,"allocated":96284672,"data_stored":96111570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721320,"kb_used_data":94100,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93650520,"statfs":{"total":96636764160,"available":95898132480,"internally_reserved":0,"allocated":96358400,"data_stored":96185298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,7,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95842304,"data_stored":95842304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95563776,"data_stored":95563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97673216,"data_stored":97673216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98398208,"data_stored":98398208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:14.478 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:15.098 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:15.099 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:15.117 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-15T21:13:14.161702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":194748962,"num_objects":23778,"num_object_clones":0,"num_object_copies":95112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161300,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2709280,"kb_used_data":381252,"kb_used_omap":14,"kb_used_meta":2327921,"kb_avail":374778080,"statfs":{"total":386547056640,"available":383772753920,"internally_reserved":0,"allocated":390402048,"data_stored":389709640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2383791666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,7,13,15,14,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18685952,"num_objects":-2281,"num_object_clones":0,"num_object_copies":-9124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2281,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004755"},"pg_stats":[{"pgid":"1.f","version":"35'10915","reported_seq":10967,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.620437+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:09.620437+0000","last_peered":"2024-09-15T21:13:09.620437+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:09.620437+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11771904,"num_objects":1437,"num_object_clones":0,"num_object_copies":5748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10915,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-839}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11108","reported_seq":16002,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:13.167877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:13.167877+0000","last_peered":"2024-09-15T21:13:13.167877+0000","last_clean":"2024-09-15T21:13:13.167877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:13.167877+0000","last_undegraded":"2024-09-15T21:13:13.167877+0000","last_fullsized":"2024-09-15T21:13:13.167877+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11812864,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10888","reported_seq":10974,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.492361+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:10.492361+0000","last_peered":"2024-09-15T21:13:10.492361+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:10.492361+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12647070,"num_objects":1545,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1545,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10887,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-713}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11022","reported_seq":11097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.776697+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:12.776697+0000","last_peered":"2024-09-15T21:13:12.776697+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:12.776697+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11744524,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11169","reported_seq":11231,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.419182+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:09.419182+0000","last_peered":"2024-09-15T21:13:09.419182+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:09.419182+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12203276,"num_objects":1490,"num_object_clones":0,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11167,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-821}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10920","reported_seq":10971,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.601339+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:09.601339+0000","last_peered":"2024-09-15T21:13:09.601339+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:09.601339+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":1456,"num_object_clones":0,"num_object_copies":5824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-860}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11297","reported_seq":11373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.588411+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:12.588411+0000","last_peered":"2024-09-15T21:13:12.588411+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:12.588411+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080396,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-891}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11279","reported_seq":11354,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.732608+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:12.732608+0000","last_peered":"2024-09-15T21:13:12.732608+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:12.732608+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137740,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'10969","reported_seq":11035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.479378+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:09.479378+0000","last_peered":"2024-09-15T21:13:09.479378+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:09.479378+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12143128,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-849}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11034","reported_seq":11120,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.583177+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:10.583177+0000","last_peered":"2024-09-15T21:13:10.583177+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:10.583177+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12514572,"num_objects":1528,"num_object_clones":0,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11032,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-807}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10900","reported_seq":15569,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:09.536451+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:09.536451+0000","last_peered":"2024-09-15T21:13:09.536451+0000","last_clean":"2024-09-15T21:13:09.536451+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:09.536451+0000","last_undegraded":"2024-09-15T21:13:09.536451+0000","last_fullsized":"2024-09-15T21:13:09.536451+0000","mapping_epoch":32,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11960320,"num_objects":1460,"num_object_clones":0,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10897","reported_seq":10943,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.619054+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:09.619054+0000","last_peered":"2024-09-15T21:13:09.619054+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:09.619054+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'800","ondisk_log_start":"27'800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12532358,"num_objects":1530,"num_object_clones":0,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-726}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11003","reported_seq":14315,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.635161+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:10.635161+0000","last_peered":"2024-09-15T21:13:10.635161+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:10.635161+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12270214,"num_objects":1498,"num_object_clones":0,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'10963","reported_seq":11026,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:09.546325+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:09.546325+0000","last_peered":"2024-09-15T21:13:09.546325+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:09.546325+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12031244,"num_objects":1469,"num_object_clones":0,"num_object_copies":5876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10961,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-810}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11104","reported_seq":11191,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:10.405257+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:10.405257+0000","last_peered":"2024-09-15T21:13:10.405257+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:10.405257+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12915980,"num_objects":1577,"num_object_clones":0,"num_object_copies":6308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11102,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-704}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11077","reported_seq":11152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.548584+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:12.548584+0000","last_peered":"2024-09-15T21:13:12.548584+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:12.548584+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":194748962,"num_objects":23778,"num_object_clones":0,"num_object_copies":95112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23778,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161300,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":387477504,"data_stored":387477504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160845,"ondisk_log_size":160845,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714428,"kb_used_data":96856,"kb_used_omap":4,"kb_used_meta":617531,"kb_avail":93657412,"statfs":{"total":96636764160,"available":95905189888,"internally_reserved":0,"allocated":99180544,"data_stored":99007442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":632352464},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":700832,"kb_used_data":96268,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":93671008,"statfs":{"total":96636764160,"available":95919112192,"internally_reserved":0,"allocated":98578432,"data_stored":98405330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572700,"kb_used_data":94028,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93799140,"statfs":{"total":96636764160,"available":96050319360,"internally_reserved":0,"allocated":96284672,"data_stored":96111570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721320,"kb_used_data":94100,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93650520,"statfs":{"total":96636764160,"available":95898132480,"internally_reserved":0,"allocated":96358400,"data_stored":96185298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,7,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95842304,"data_stored":95842304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95563776,"data_stored":95563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":97673216,"data_stored":97673216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98398208,"data_stored":98398208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:15.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:15.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'10915", 'reported_seq': 10967, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.620437+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:09.620437+0000', 'last_peered': '2024-09-15T21:13:09.620437+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:09.620437+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11771904, 'num_objects': 1437, 'num_object_clones': 0, 'num_object_copies': 5748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1437, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10915, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -839}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.118 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:15.118 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10888", 'reported_seq': 10974, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:10.492361+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:10.492361+0000', 'last_peered': '2024-09-15T21:13:10.492361+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:10.492361+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10088, 'log_dups_size': 0, 'ondisk_log_size': 10088, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12647070, 'num_objects': 1545, 'num_object_clones': 0, 'num_object_copies': 6180, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1545, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10887, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -713}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11022", 'reported_seq': 11097, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:12.776697+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:12.776697+0000', 'last_peered': '2024-09-15T21:13:12.776697+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:12.776697+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11744524, 'num_objects': 1434, 'num_object_clones': 0, 'num_object_copies': 5736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1434, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11020, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -900}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11169", 'reported_seq': 11231, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.419182+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:09.419182+0000', 'last_peered': '2024-09-15T21:13:09.419182+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:09.419182+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12203276, 'num_objects': 1490, 'num_object_clones': 0, 'num_object_copies': 5960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11167, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -821}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'10920", 'reported_seq': 10971, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.601339+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:09.601339+0000', 'last_peered': '2024-09-15T21:13:09.601339+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:09.601339+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11927552, 'num_objects': 1456, 'num_object_clones': 0, 'num_object_copies': 5824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1456, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10920, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -860}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11297", 'reported_seq': 11373, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:12.588411+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:12.588411+0000', 'last_peered': '2024-09-15T21:13:12.588411+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:12.588411+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12080396, 'num_objects': 1475, 'num_object_clones': 0, 'num_object_copies': 5900, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1475, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11295, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -891}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11279", 'reported_seq': 11354, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:12.732608+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:12.732608+0000', 'last_peered': '2024-09-15T21:13:12.732608+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:12.732608+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12137740, 'num_objects': 1482, 'num_object_clones': 0, 'num_object_copies': 5928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11277, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -992}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:15.119 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'10969", 'reported_seq': 11035, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.479378+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:09.479378+0000', 'last_peered': '2024-09-15T21:13:09.479378+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:09.479378+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12143128, 'num_objects': 1483, 'num_object_clones': 0, 'num_object_copies': 5932, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1483, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10966, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -849}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11034", 'reported_seq': 11120, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:10.583177+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:10.583177+0000', 'last_peered': '2024-09-15T21:13:10.583177+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:10.583177+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12514572, 'num_objects': 1528, 'num_object_clones': 0, 'num_object_copies': 6112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1528, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11032, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -807}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'10897", 'reported_seq': 10943, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.619054+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:09.619054+0000', 'last_peered': '2024-09-15T21:13:09.619054+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:09.619054+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'800", 'ondisk_log_start': "27'800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12532358, 'num_objects': 1530, 'num_object_clones': 0, 'num_object_copies': 6120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1530, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10896, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11003", 'reported_seq': 14315, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:10.635161+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:10.635161+0000', 'last_peered': '2024-09-15T21:13:10.635161+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:10.635161+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12270214, 'num_objects': 1498, 'num_object_clones': 0, 'num_object_copies': 5992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 674, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1498, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11002, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1614, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'10963", 'reported_seq': 11026, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:09.546325+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:09.546325+0000', 'last_peered': '2024-09-15T21:13:09.546325+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:09.546325+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12031244, 'num_objects': 1469, 'num_object_clones': 0, 'num_object_copies': 5876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1469, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 10961, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -810}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:15.120 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11104", 'reported_seq': 11191, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:10.405257+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:10.405257+0000', 'last_peered': '2024-09-15T21:13:10.405257+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:10.405257+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12915980, 'num_objects': 1577, 'num_object_clones': 0, 'num_object_copies': 6308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11102, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:15.121 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:15.121 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11077", 'reported_seq': 11152, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:12.548584+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:12.548584+0000', 'last_peered': '2024-09-15T21:13:12.548584+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:12.548584+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12055820, 'num_objects': 1472, 'num_object_clones': 0, 'num_object_copies': 5888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1472, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11075, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -843}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:15.121 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:15.911 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:15.911 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30828/95112 objects degraded (32.412%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":661,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":23778,"data_bytes":194748962,"bytes_used":2774302720,"bytes_avail":383772753920,"bytes_total":386547056640,"degraded_objects":30828,"degraded_total":95112,"degraded_ratio":0.32412313903608381,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:15.941 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:15.941 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:16.569 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:16.569 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:16.592 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":357,"stamp":"2024-09-15T21:13:16.162405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":189301282,"num_objects":23113,"num_object_clones":0,"num_object_copies":92452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161965,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160910,"ondisk_log_size":160910,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2711036,"kb_used_data":373296,"kb_used_omap":14,"kb_used_meta":2337649,"kb_avail":374776324,"statfs":{"total":386547056640,"available":383770955776,"internally_reserved":0,"allocated":382255104,"data_stored":381562696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2393753138},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,9,14,17,10,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21815296,"num_objects":-2663,"num_object_clones":0,"num_object_copies":-10652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004649"},"pg_stats":[{"pgid":"1.f","version":"35'10963","reported_seq":11015,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.328592+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:14.328592+0000","last_peered":"2024-09-15T21:13:14.328592+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:14.328592+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11378688,"num_objects":1389,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-887}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11108","reported_seq":16002,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:13.167877+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:13.167877+0000","last_peered":"2024-09-15T21:13:13.167877+0000","last_clean":"2024-09-15T21:13:13.167877+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:13.167877+0000","last_undegraded":"2024-09-15T21:13:13.167877+0000","last_fullsized":"2024-09-15T21:13:13.167877+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11812864,"num_objects":1442,"num_object_clones":0,"num_object_copies":5768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1774,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10952","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.700598+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:15.700598+0000","last_peered":"2024-09-15T21:13:15.700598+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:15.700598+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12122782,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10951,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11022","reported_seq":11097,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.776697+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:12.776697+0000","last_peered":"2024-09-15T21:13:12.776697+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:12.776697+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11744524,"num_objects":1434,"num_object_clones":0,"num_object_copies":5736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-900}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11230","reported_seq":11292,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.677405+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:14.677405+0000","last_peered":"2024-09-15T21:13:14.677405+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:14.677405+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11703564,"num_objects":1429,"num_object_clones":0,"num_object_copies":5716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11228,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-882}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'10984","reported_seq":11035,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.559038+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:14.559038+0000","last_peered":"2024-09-15T21:13:14.559038+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:14.559038+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":1392,"num_object_clones":0,"num_object_copies":5568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-924}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11297","reported_seq":11373,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.588411+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:12.588411+0000","last_peered":"2024-09-15T21:13:12.588411+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:12.588411+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12080396,"num_objects":1475,"num_object_clones":0,"num_object_copies":5900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11295,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-891}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11279","reported_seq":11354,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.732608+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:12.732608+0000","last_peered":"2024-09-15T21:13:12.732608+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:12.732608+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12137740,"num_objects":1482,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11277,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-992}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11023","reported_seq":11089,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.677081+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:14.677081+0000","last_peered":"2024-09-15T21:13:14.677081+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:14.677081+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11700760,"num_objects":1429,"num_object_clones":0,"num_object_copies":5716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11020,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-903}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11097","reported_seq":11185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.547745+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:15.547745+0000","last_peered":"2024-09-15T21:13:15.547745+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:15.547745+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11998476,"num_objects":1465,"num_object_clones":0,"num_object_copies":5860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11095,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-870}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'10958","reported_seq":15627,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:14.533987+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:14.533987+0000","last_peered":"2024-09-15T21:13:14.533987+0000","last_clean":"2024-09-15T21:13:14.533987+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:14.533987+0000","last_undegraded":"2024-09-15T21:13:14.533987+0000","last_fullsized":"2024-09-15T21:13:14.533987+0000","mapping_epoch":32,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11485184,"num_objects":1402,"num_object_clones":0,"num_object_copies":5608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5069,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'10944","reported_seq":10990,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.699883+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:14.699883+0000","last_peered":"2024-09-15T21:13:14.699883+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:14.699883+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12147334,"num_objects":1483,"num_object_clones":0,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10943,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-773}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11076","reported_seq":14503,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.653017+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:15.653017+0000","last_peered":"2024-09-15T21:13:15.653017+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:15.653017+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11672198,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11037","reported_seq":11100,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:14.564278+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:14.564278+0000","last_peered":"2024-09-15T21:13:14.564278+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:14.564278+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11425036,"num_objects":1395,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11035,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-884}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11163","reported_seq":11252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.629721+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:15.629721+0000","last_peered":"2024-09-15T21:13:15.629721+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:15.629721+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12432652,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11077","reported_seq":11152,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:12.548584+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:12.548584+0000","last_peered":"2024-09-15T21:13:12.548584+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:12.548584+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12055820,"num_objects":1472,"num_object_clones":0,"num_object_copies":5888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-843}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":189301282,"num_objects":23113,"num_object_clones":0,"num_object_copies":92452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23113,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161965,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":379338752,"data_stored":379338752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160910,"ondisk_log_size":160910,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":697968,"kb_used_data":92888,"kb_used_omap":4,"kb_used_meta":605051,"kb_avail":93673872,"statfs":{"total":96636764160,"available":95922044928,"internally_reserved":0,"allocated":95117312,"data_stored":94944210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":619572944},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,5,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719048,"kb_used_data":92280,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93652792,"statfs":{"total":96636764160,"available":95900459008,"internally_reserved":0,"allocated":94494720,"data_stored":94321618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051695,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":572700,"kb_used_data":94028,"kb_used_omap":4,"kb_used_meta":478651,"kb_avail":93799140,"statfs":{"total":96636764160,"available":96050319360,"internally_reserved":0,"allocated":96284672,"data_stored":96111570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":490139343},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247933,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721320,"kb_used_data":94100,"kb_used_omap":4,"kb_used_meta":627195,"kb_avail":93650520,"statfs":{"total":96636764160,"available":95898132480,"internally_reserved":0,"allocated":96358400,"data_stored":96185298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":642248398},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,7,6,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":95842304,"data_stored":95842304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":95563776,"data_stored":95563776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93601792,"data_stored":93601792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94330880,"data_stored":94330880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:19.595 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:20.185 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:20.186 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:20.212 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-15T21:13:20.163830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183419426,"num_objects":22395,"num_object_clones":0,"num_object_copies":89580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162683,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766008,"kb_used_data":361636,"kb_used_omap":14,"kb_used_meta":2404273,"kb_avail":374721352,"statfs":{"total":386547056640,"available":383714664448,"internally_reserved":0,"allocated":370315264,"data_stored":369622856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2461976114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,19,13,8,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19963904,"num_objects":-2437,"num_object_clones":0,"num_object_copies":-9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004469"},"pg_stats":[{"pgid":"1.f","version":"35'11016","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.851997+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:19.851997+0000","last_peered":"2024-09-15T21:13:19.851997+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:19.851997+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11164","reported_seq":16058,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:18.233519+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:18.233519+0000","last_peered":"2024-09-15T21:13:18.233519+0000","last_clean":"2024-09-15T21:13:18.233519+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:18.233519+0000","last_undegraded":"2024-09-15T21:13:18.233519+0000","last_fullsized":"2024-09-15T21:13:18.233519+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11354112,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10952","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.700598+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:15.700598+0000","last_peered":"2024-09-15T21:13:15.700598+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:15.700598+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12122782,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10951,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11084","reported_seq":11160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.769297+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:17.769297+0000","last_peered":"2024-09-15T21:13:17.769297+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:17.769297+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236620,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11291","reported_seq":11353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.791044+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:19.791044+0000","last_peered":"2024-09-15T21:13:19.791044+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:19.791044+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203852,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11054","reported_seq":11105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.682297+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:19.682297+0000","last_peered":"2024-09-15T21:13:19.682297+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:19.682297+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10829824,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11364","reported_seq":11440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.860401+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:17.860401+0000","last_peered":"2024-09-15T21:13:17.860401+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:17.860401+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11531532,"num_objects":1408,"num_object_clones":0,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11348","reported_seq":11423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.804859+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:17.804859+0000","last_peered":"2024-09-15T21:13:17.804859+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:17.804859+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11572492,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11083","reported_seq":11149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.807176+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:19.807176+0000","last_peered":"2024-09-15T21:13:19.807176+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:19.807176+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11209240,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11097","reported_seq":11185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.547745+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:15.547745+0000","last_peered":"2024-09-15T21:13:15.547745+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:15.547745+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11998476,"num_objects":1465,"num_object_clones":0,"num_object_copies":5860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11095,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-870}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11001","reported_seq":15670,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:19.590205+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:19.590205+0000","last_peered":"2024-09-15T21:13:19.590205+0000","last_clean":"2024-09-15T21:13:19.590205+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:19.590205+0000","last_undegraded":"2024-09-15T21:13:19.590205+0000","last_fullsized":"2024-09-15T21:13:19.590205+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11132928,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11003","reported_seq":11049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.620776+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:19.620776+0000","last_peered":"2024-09-15T21:13:19.620776+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:19.620776+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664006,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11076","reported_seq":14503,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.653017+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:15.653017+0000","last_peered":"2024-09-15T21:13:15.653017+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:15.653017+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11672198,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.625337+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:19.625337+0000","last_peered":"2024-09-15T21:13:19.625337+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:19.625337+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11163","reported_seq":11252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.629721+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:15.629721+0000","last_peered":"2024-09-15T21:13:15.629721+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:15.629721+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12432652,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11145","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.535419+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:17.535419+0000","last_peered":"2024-09-15T21:13:17.535419+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:17.535419+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498764,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":183419426,"num_objects":22395,"num_object_clones":0,"num_object_copies":89580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162683,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":367263744,"data_stored":367263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716348,"kb_used_data":89048,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93655492,"statfs":{"total":96636764160,"available":95903223808,"internally_reserved":0,"allocated":91185152,"data_stored":91012050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719048,"kb_used_data":92280,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93652792,"statfs":{"total":96636764160,"available":95900459008,"internally_reserved":0,"allocated":94494720,"data_stored":94321618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591008,"kb_used_data":90128,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93780832,"statfs":{"total":96636764160,"available":96031571968,"internally_reserved":0,"allocated":92291072,"data_stored":92117970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739604,"kb_used_data":90180,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93632236,"statfs":{"total":96636764160,"available":95879409664,"internally_reserved":0,"allocated":92344320,"data_stored":92171218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,2,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91873280,"data_stored":91873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91512832,"data_stored":91512832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93601792,"data_stored":93601792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90275840,"data_stored":90275840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:20.213 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:20.867 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:20.868 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:20.893 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-09-15T21:13:20.163830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183419426,"num_objects":22395,"num_object_clones":0,"num_object_copies":89580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162683,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766008,"kb_used_data":361636,"kb_used_omap":14,"kb_used_meta":2404273,"kb_avail":374721352,"statfs":{"total":386547056640,"available":383714664448,"internally_reserved":0,"allocated":370315264,"data_stored":369622856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2461976114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,8,19,13,8,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19963904,"num_objects":-2437,"num_object_clones":0,"num_object_copies":-9748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2437,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004469"},"pg_stats":[{"pgid":"1.f","version":"35'11016","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.851997+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:19.851997+0000","last_peered":"2024-09-15T21:13:19.851997+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:19.851997+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11164","reported_seq":16058,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:18.233519+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:18.233519+0000","last_peered":"2024-09-15T21:13:18.233519+0000","last_clean":"2024-09-15T21:13:18.233519+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:18.233519+0000","last_undegraded":"2024-09-15T21:13:18.233519+0000","last_fullsized":"2024-09-15T21:13:18.233519+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11354112,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'10952","reported_seq":11038,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.700598+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:15.700598+0000","last_peered":"2024-09-15T21:13:15.700598+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:15.700598+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'900","ondisk_log_start":"27'900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12122782,"num_objects":1481,"num_object_clones":0,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1481,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10951,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-777}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11084","reported_seq":11160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.769297+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:17.769297+0000","last_peered":"2024-09-15T21:13:17.769297+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:17.769297+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236620,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11291","reported_seq":11353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.791044+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:19.791044+0000","last_peered":"2024-09-15T21:13:19.791044+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:19.791044+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203852,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11054","reported_seq":11105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.682297+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:19.682297+0000","last_peered":"2024-09-15T21:13:19.682297+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:19.682297+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10829824,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11364","reported_seq":11440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.860401+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:17.860401+0000","last_peered":"2024-09-15T21:13:17.860401+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:17.860401+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11531532,"num_objects":1408,"num_object_clones":0,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11348","reported_seq":11423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.804859+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:17.804859+0000","last_peered":"2024-09-15T21:13:17.804859+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:17.804859+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11572492,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11083","reported_seq":11149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.807176+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:19.807176+0000","last_peered":"2024-09-15T21:13:19.807176+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:19.807176+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11209240,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11097","reported_seq":11185,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.547745+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:15.547745+0000","last_peered":"2024-09-15T21:13:15.547745+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:15.547745+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11998476,"num_objects":1465,"num_object_clones":0,"num_object_copies":5860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11095,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-870}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11001","reported_seq":15670,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:19.590205+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:19.590205+0000","last_peered":"2024-09-15T21:13:19.590205+0000","last_clean":"2024-09-15T21:13:19.590205+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:19.590205+0000","last_undegraded":"2024-09-15T21:13:19.590205+0000","last_fullsized":"2024-09-15T21:13:19.590205+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11132928,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11003","reported_seq":11049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.620776+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:19.620776+0000","last_peered":"2024-09-15T21:13:19.620776+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:19.620776+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664006,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11076","reported_seq":14503,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.653017+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:15.653017+0000","last_peered":"2024-09-15T21:13:15.653017+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:15.653017+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11672198,"num_objects":1425,"num_object_clones":0,"num_object_copies":5700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11075,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1671,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":808}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.625337+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:19.625337+0000","last_peered":"2024-09-15T21:13:19.625337+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:19.625337+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11163","reported_seq":11252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:15.629721+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:15.629721+0000","last_peered":"2024-09-15T21:13:15.629721+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:15.629721+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12432652,"num_objects":1518,"num_object_clones":0,"num_object_copies":6072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11145","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.535419+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:17.535419+0000","last_peered":"2024-09-15T21:13:17.535419+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:17.535419+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498764,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":183419426,"num_objects":22395,"num_object_clones":0,"num_object_copies":89580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22395,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162683,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1692,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":367263744,"data_stored":367263744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160928,"ondisk_log_size":160928,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716348,"kb_used_data":89048,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93655492,"statfs":{"total":96636764160,"available":95903223808,"internally_reserved":0,"allocated":91185152,"data_stored":91012050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719048,"kb_used_data":92280,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":93652792,"statfs":{"total":96636764160,"available":95900459008,"internally_reserved":0,"allocated":94494720,"data_stored":94321618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":641792453},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,5,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591008,"kb_used_data":90128,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93780832,"statfs":{"total":96636764160,"available":96031571968,"internally_reserved":0,"allocated":92291072,"data_stored":92117970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739604,"kb_used_data":90180,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93632236,"statfs":{"total":96636764160,"available":95879409664,"internally_reserved":0,"allocated":92344320,"data_stored":92171218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,2,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91873280,"data_stored":91873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91512832,"data_stored":91512832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":93601792,"data_stored":93601792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90275840,"data_stored":90275840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:20.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:20.894 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11016", 'reported_seq': 11067, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.851997+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:19.851997+0000', 'last_peered': '2024-09-15T21:13:19.851997+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:19.851997+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10016, 'log_dups_size': 0, 'ondisk_log_size': 10016, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10944512, 'num_objects': 1336, 'num_object_clones': 0, 'num_object_copies': 5344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1336, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11016, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -940}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'10952", 'reported_seq': 11038, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:15.700598+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:15.700598+0000', 'last_peered': '2024-09-15T21:13:15.700598+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:15.700598+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'900", 'ondisk_log_start': "27'900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12122782, 'num_objects': 1481, 'num_object_clones': 0, 'num_object_copies': 5924, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1481, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 10951, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -777}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11084", 'reported_seq': 11160, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:17.769297+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:17.769297+0000', 'last_peered': '2024-09-15T21:13:17.769297+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:17.769297+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11236620, 'num_objects': 1372, 'num_object_clones': 0, 'num_object_copies': 5488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1372, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11082, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -962}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11291", 'reported_seq': 11353, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.791044+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:19.791044+0000', 'last_peered': '2024-09-15T21:13:19.791044+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:19.791044+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11203852, 'num_objects': 1368, 'num_object_clones': 0, 'num_object_copies': 5472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1368, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11289, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -943}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11054", 'reported_seq': 11105, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.682297+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:19.682297+0000', 'last_peered': '2024-09-15T21:13:19.682297+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:19.682297+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10829824, 'num_objects': 1322, 'num_object_clones': 0, 'num_object_copies': 5288, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1322, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11054, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -994}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:20.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11364", 'reported_seq': 11440, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:17.860401+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:17.860401+0000', 'last_peered': '2024-09-15T21:13:17.860401+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:17.860401+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11531532, 'num_objects': 1408, 'num_object_clones': 0, 'num_object_copies': 5632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1408, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11362, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -958}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11348", 'reported_seq': 11423, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:17.804859+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:17.804859+0000', 'last_peered': '2024-09-15T21:13:17.804859+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:17.804859+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11572492, 'num_objects': 1413, 'num_object_clones': 0, 'num_object_copies': 5652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1413, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1061}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11083", 'reported_seq': 11149, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.807176+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:19.807176+0000', 'last_peered': '2024-09-15T21:13:19.807176+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:19.807176+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11209240, 'num_objects': 1369, 'num_object_clones': 0, 'num_object_copies': 5476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11080, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -963}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11097", 'reported_seq': 11185, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:15.547745+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:15.547745+0000', 'last_peered': '2024-09-15T21:13:15.547745+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:15.547745+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11998476, 'num_objects': 1465, 'num_object_clones': 0, 'num_object_copies': 5860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1465, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11095, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -870}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11003", 'reported_seq': 11049, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.620776+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:19.620776+0000', 'last_peered': '2024-09-15T21:13:19.620776+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:19.620776+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11664006, 'num_objects': 1424, 'num_object_clones': 0, 'num_object_copies': 5696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1424, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11002, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -832}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11076", 'reported_seq': 14503, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:15.653017+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:15.653017+0000', 'last_peered': '2024-09-15T21:13:15.653017+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:15.653017+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11672198, 'num_objects': 1425, 'num_object_clones': 0, 'num_object_copies': 5700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 617, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1425, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11075, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1671, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 808}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11087", 'reported_seq': 11150, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:19.625337+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:19.625337+0000', 'last_peered': '2024-09-15T21:13:19.625337+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:19.625337+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11015436, 'num_objects': 1345, 'num_object_clones': 0, 'num_object_copies': 5380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1345, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11085, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -934}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11163", 'reported_seq': 11252, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:15.629721+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:15.629721+0000', 'last_peered': '2024-09-15T21:13:15.629721+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:15.629721+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12432652, 'num_objects': 1518, 'num_object_clones': 0, 'num_object_copies': 6072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1518, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11161, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:20.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11145", 'reported_seq': 11220, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:17.535419+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:17.535419+0000', 'last_peered': '2024-09-15T21:13:17.535419+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:17.535419+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11498764, 'num_objects': 1404, 'num_object_clones': 0, 'num_object_copies': 5616, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1404, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11143, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -911}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:20.897 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:21.620 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:21.621 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30771/89580 objects degraded (34.350%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":667,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22395,"data_bytes":183419426,"bytes_used":2832392192,"bytes_avail":383714664448,"bytes_total":386547056640,"degraded_objects":30771,"degraded_total":89580,"degraded_ratio":0.34350301406563966,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":203},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:21.641 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:21.641 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:22.296 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:22.296 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:22.331 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-15T21:13:22.164752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":181166626,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162958,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2766348,"kb_used_data":357816,"kb_used_omap":14,"kb_used_meta":2408433,"kb_avail":374721012,"statfs":{"total":386547056640,"available":383714316288,"internally_reserved":0,"allocated":366403584,"data_stored":365711176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2466235954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,15,10,10,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18169856,"num_objects":-2218,"num_object_clones":0,"num_object_copies":-8872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004704"},"pg_stats":[{"pgid":"1.f","version":"35'11016","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.851997+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:19.851997+0000","last_peered":"2024-09-15T21:13:19.851997+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:19.851997+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11164","reported_seq":16058,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:18.233519+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:18.233519+0000","last_peered":"2024-09-15T21:13:18.233519+0000","last_clean":"2024-09-15T21:13:18.233519+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:18.233519+0000","last_undegraded":"2024-09-15T21:13:18.233519+0000","last_fullsized":"2024-09-15T21:13:18.233519+0000","mapping_epoch":34,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11354112,"num_objects":1386,"num_object_clones":0,"num_object_copies":5544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11034","reported_seq":11117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.665521+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:20.665521+0000","last_peered":"2024-09-15T21:13:20.665521+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:20.665521+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11451038,"num_objects":1399,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11033,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11084","reported_seq":11160,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.769297+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:17.769297+0000","last_peered":"2024-09-15T21:13:17.769297+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:17.769297+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11236620,"num_objects":1372,"num_object_clones":0,"num_object_copies":5488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11082,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-962}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11291","reported_seq":11353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.791044+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:19.791044+0000","last_peered":"2024-09-15T21:13:19.791044+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:19.791044+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203852,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11054","reported_seq":11105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.682297+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:19.682297+0000","last_peered":"2024-09-15T21:13:19.682297+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:19.682297+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10829824,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11364","reported_seq":11440,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.860401+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:17.860401+0000","last_peered":"2024-09-15T21:13:17.860401+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:17.860401+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11531532,"num_objects":1408,"num_object_clones":0,"num_object_copies":5632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11362,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-958}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11348","reported_seq":11423,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.804859+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:17.804859+0000","last_peered":"2024-09-15T21:13:17.804859+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:17.804859+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11572492,"num_objects":1413,"num_object_clones":0,"num_object_copies":5652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1061}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11083","reported_seq":11149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.807176+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:19.807176+0000","last_peered":"2024-09-15T21:13:19.807176+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:19.807176+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11209240,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11167","reported_seq":11252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.685629+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:20.685629+0000","last_peered":"2024-09-15T21:13:20.685629+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:20.685629+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11425036,"num_objects":1395,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11001","reported_seq":15670,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:19.590205+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:19.590205+0000","last_peered":"2024-09-15T21:13:19.590205+0000","last_clean":"2024-09-15T21:13:19.590205+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:19.590205+0000","last_undegraded":"2024-09-15T21:13:19.590205+0000","last_fullsized":"2024-09-15T21:13:19.590205+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11132928,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11003","reported_seq":11049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.620776+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:19.620776+0000","last_peered":"2024-09-15T21:13:19.620776+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:19.620776+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664006,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11135","reported_seq":14689,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.754077+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:20.754077+0000","last_peered":"2024-09-15T21:13:20.754077+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:20.754077+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11188870,"num_objects":1366,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.625337+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:19.625337+0000","last_peered":"2024-09-15T21:13:19.625337+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:19.625337+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11227","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.685348+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:20.685348+0000","last_peered":"2024-09-15T21:13:20.685348+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:20.685348+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908364,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11145","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:17.535419+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:17.535419+0000","last_peered":"2024-09-15T21:13:17.535419+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:17.535419+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11498764,"num_objects":1404,"num_object_clones":0,"num_object_copies":5616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-911}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":181166626,"num_objects":22120,"num_object_clones":0,"num_object_copies":88480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22120,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162958,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":363290624,"data_stored":363290624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716348,"kb_used_data":89048,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93655492,"statfs":{"total":96636764160,"available":95903223808,"internally_reserved":0,"allocated":91185152,"data_stored":91012050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719388,"kb_used_data":88460,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93652452,"statfs":{"total":96636764160,"available":95900110848,"internally_reserved":0,"allocated":90583040,"data_stored":90409938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051696,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591008,"kb_used_data":90128,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93780832,"statfs":{"total":96636764160,"available":96031571968,"internally_reserved":0,"allocated":92291072,"data_stored":92117970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,8,2,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247934,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739604,"kb_used_data":90180,"kb_used_omap":4,"kb_used_meta":649403,"kb_avail":93632236,"statfs":{"total":96636764160,"available":95879409664,"internally_reserved":0,"allocated":92344320,"data_stored":92171218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":664989390},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,5,2,3,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91873280,"data_stored":91873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":91512832,"data_stored":91512832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89628672,"data_stored":89628672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90275840,"data_stored":90275840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:25.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:26.016 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:26.016 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:26.046 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-09-15T21:13:24.165514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":178741794,"num_objects":21824,"num_object_clones":0,"num_object_copies":87296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163254,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767144,"kb_used_data":350244,"kb_used_omap":14,"kb_used_meta":2416753,"kb_avail":374720216,"statfs":{"total":386547056640,"available":383713501184,"internally_reserved":0,"allocated":358649856,"data_stored":357957448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2474755634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,3,4,17,10,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18530304,"num_objects":-2262,"num_object_clones":0,"num_object_copies":-9048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004549"},"pg_stats":[{"pgid":"1.f","version":"35'11016","reported_seq":11067,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.851997+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:19.851997+0000","last_peered":"2024-09-15T21:13:19.851997+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:19.851997+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":1336,"num_object_clones":0,"num_object_copies":5344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11016,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11224","reported_seq":16118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:23.264573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:23.264573+0000","last_peered":"2024-09-15T21:13:23.264573+0000","last_clean":"2024-09-15T21:13:23.264573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:23.264573+0000","last_undegraded":"2024-09-15T21:13:23.264573+0000","last_fullsized":"2024-09-15T21:13:23.264573+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10862592,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11034","reported_seq":11117,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.665521+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:20.665521+0000","last_peered":"2024-09-15T21:13:20.665521+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:20.665521+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11451038,"num_objects":1399,"num_object_clones":0,"num_object_copies":5596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11033,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-859}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11146","reported_seq":11222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.573715+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:22.573715+0000","last_peered":"2024-09-15T21:13:22.573715+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:22.573715+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10728716,"num_objects":1310,"num_object_clones":0,"num_object_copies":5240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11291","reported_seq":11353,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.791044+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:19.791044+0000","last_peered":"2024-09-15T21:13:19.791044+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:19.791044+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11203852,"num_objects":1368,"num_object_clones":0,"num_object_copies":5472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11289,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-943}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11054","reported_seq":11105,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.682297+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:19.682297+0000","last_peered":"2024-09-15T21:13:19.682297+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:19.682297+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10829824,"num_objects":1322,"num_object_clones":0,"num_object_copies":5288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11054,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-994}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11420","reported_seq":11496,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.766091+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:22.766091+0000","last_peered":"2024-09-15T21:13:22.766091+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:22.766091+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11072780,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1014}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11411","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.715735+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:22.715735+0000","last_peered":"2024-09-15T21:13:22.715735+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:22.715735+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11409,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11083","reported_seq":11149,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.807176+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:19.807176+0000","last_peered":"2024-09-15T21:13:19.807176+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:19.807176+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11209240,"num_objects":1369,"num_object_clones":0,"num_object_copies":5476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11167","reported_seq":11252,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.685629+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:20.685629+0000","last_peered":"2024-09-15T21:13:20.685629+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:20.685629+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11425036,"num_objects":1395,"num_object_clones":0,"num_object_copies":5580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11165,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-940}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11001","reported_seq":15670,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:19.590205+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:19.590205+0000","last_peered":"2024-09-15T21:13:19.590205+0000","last_clean":"2024-09-15T21:13:19.590205+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:19.590205+0000","last_undegraded":"2024-09-15T21:13:19.590205+0000","last_fullsized":"2024-09-15T21:13:19.590205+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11132928,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5112,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11003","reported_seq":11049,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.620776+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:19.620776+0000","last_peered":"2024-09-15T21:13:19.620776+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:19.620776+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11664006,"num_objects":1424,"num_object_clones":0,"num_object_copies":5696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-832}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11135","reported_seq":14689,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.754077+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:20.754077+0000","last_peered":"2024-09-15T21:13:20.754077+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:20.754077+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11188870,"num_objects":1366,"num_object_clones":0,"num_object_copies":5464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1735,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11087","reported_seq":11150,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:19.625337+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:19.625337+0000","last_peered":"2024-09-15T21:13:19.625337+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:19.625337+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11015436,"num_objects":1345,"num_object_clones":0,"num_object_copies":5380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11085,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-934}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11227","reported_seq":11316,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:20.685348+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:20.685348+0000","last_peered":"2024-09-15T21:13:20.685348+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:20.685348+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11908364,"num_objects":1454,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11225,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-827}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11200","reported_seq":11274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.763536+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:22.763536+0000","last_peered":"2024-09-15T21:13:22.763536+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:22.763536+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11048204,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":178741794,"num_objects":21824,"num_object_clones":0,"num_object_copies":87296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30707,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21824,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163254,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":355500032,"data_stored":355500032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160699,"ondisk_log_size":160699,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716348,"kb_used_data":89048,"kb_used_omap":4,"kb_used_meta":627259,"kb_avail":93655492,"statfs":{"total":96636764160,"available":95903223808,"internally_reserved":0,"allocated":91185152,"data_stored":91012050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":642313936},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719388,"kb_used_data":88460,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93652452,"statfs":{"total":96636764160,"available":95900110848,"internally_reserved":0,"allocated":90583040,"data_stored":90409938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1,2,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":86356,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":88428544,"data_stored":88255442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739972,"kb_used_data":86380,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93631868,"statfs":{"total":96636764160,"available":95879032832,"internally_reserved":0,"allocated":88453120,"data_stored":88280018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,9,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88006656,"data_stored":88006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87588864,"data_stored":87588864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":89628672,"data_stored":89628672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90275840,"data_stored":90275840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:26.048 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:26.677 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:26.677 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:26.710 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-15T21:13:26.166223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173318690,"num_objects":21162,"num_object_clones":0,"num_object_copies":84648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163916,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767792,"kb_used_data":342564,"kb_used_omap":14,"kb_used_meta":2425073,"kb_avail":374719568,"statfs":{"total":386547056640,"available":383712837632,"internally_reserved":0,"allocated":350785536,"data_stored":350093128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2483275314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,7,12,19,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21430272,"num_objects":-2616,"num_object_clones":0,"num_object_copies":-10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004520"},"pg_stats":[{"pgid":"1.f","version":"35'11080","reported_seq":11131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.986868+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:24.986868+0000","last_peered":"2024-09-15T21:13:24.986868+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:24.986868+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11224","reported_seq":16118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:23.264573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:23.264573+0000","last_peered":"2024-09-15T21:13:23.264573+0000","last_clean":"2024-09-15T21:13:23.264573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:23.264573+0000","last_undegraded":"2024-09-15T21:13:23.264573+0000","last_fullsized":"2024-09-15T21:13:23.264573+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10862592,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11094","reported_seq":11180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.698479+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:25.698479+0000","last_peered":"2024-09-15T21:13:25.698479+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:25.698479+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10959518,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11093,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11146","reported_seq":11222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.573715+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:22.573715+0000","last_peered":"2024-09-15T21:13:22.573715+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:22.573715+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10728716,"num_objects":1310,"num_object_clones":0,"num_object_copies":5240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11354","reported_seq":11417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.918925+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:24.918925+0000","last_peered":"2024-09-15T21:13:24.918925+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:24.918925+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10687756,"num_objects":1305,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11107","reported_seq":11158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.933310+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:24.933310+0000","last_peered":"2024-09-15T21:13:24.933310+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:24.933310+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10395648,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11420","reported_seq":11496,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.766091+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:22.766091+0000","last_peered":"2024-09-15T21:13:22.766091+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:22.766091+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11072780,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1014}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11411","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.715735+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:22.715735+0000","last_peered":"2024-09-15T21:13:22.715735+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:22.715735+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11409,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11138","reported_seq":11203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.811988+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:24.811988+0000","last_peered":"2024-09-15T21:13:24.811988+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:24.811988+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10758680,"num_objects":1314,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11135,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11241","reported_seq":11328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.871363+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:25.871363+0000","last_peered":"2024-09-15T21:13:25.871363+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:25.871363+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10818828,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11068","reported_seq":15737,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:24.947293+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:24.947293+0000","last_peered":"2024-09-15T21:13:24.947293+0000","last_clean":"2024-09-15T21:13:24.947293+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:24.947293+0000","last_undegraded":"2024-09-15T21:13:24.947293+0000","last_fullsized":"2024-09-15T21:13:24.947293+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11068","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.836680+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:24.836680+0000","last_peered":"2024-09-15T21:13:24.836680+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:24.836680+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11131526,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11194","reported_seq":14866,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867696+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:25.867696+0000","last_peered":"2024-09-15T21:13:25.867696+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:25.867696+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11142","reported_seq":11205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.706602+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:24.706602+0000","last_peered":"2024-09-15T21:13:24.706602+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:24.706602+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10564876,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11274","reported_seq":11363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867572+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:25.867572+0000","last_peered":"2024-09-15T21:13:25.867572+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:25.867572+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523340,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11200","reported_seq":11274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.763536+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:22.763536+0000","last_peered":"2024-09-15T21:13:22.763536+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:22.763536+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11048204,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173318690,"num_objects":21162,"num_object_clones":0,"num_object_copies":84648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163916,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347394048,"data_stored":347394048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716656,"kb_used_data":85188,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93655184,"statfs":{"total":96636764160,"available":95902908416,"internally_reserved":0,"allocated":87232512,"data_stored":87059410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719728,"kb_used_data":84640,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93652112,"statfs":{"total":96636764160,"available":95899762688,"internally_reserved":0,"allocated":86671360,"data_stored":86498258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":86356,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":88428544,"data_stored":88255442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739972,"kb_used_data":86380,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93631868,"statfs":{"total":96636764160,"available":95879032832,"internally_reserved":0,"allocated":88453120,"data_stored":88280018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,9,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88006656,"data_stored":88006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87588864,"data_stored":87588864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85528576,"data_stored":85528576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86269952,"data_stored":86269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:26.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:26.711 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11080", 'reported_seq': 11131, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.986868+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:24.986868+0000', 'last_peered': '2024-09-15T21:13:24.986868+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:24.986868+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10420224, 'num_objects': 1272, 'num_object_clones': 0, 'num_object_copies': 5088, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1272, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11080, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1004}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.711 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11094", 'reported_seq': 11180, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:25.698479+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:25.698479+0000', 'last_peered': '2024-09-15T21:13:25.698479+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:25.698479+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10959518, 'num_objects': 1339, 'num_object_clones': 0, 'num_object_copies': 5356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1339, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11093, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -919}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11146", 'reported_seq': 11222, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:22.573715+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:22.573715+0000', 'last_peered': '2024-09-15T21:13:22.573715+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:22.573715+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10728716, 'num_objects': 1310, 'num_object_clones': 0, 'num_object_copies': 5240, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1310, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11144, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11354", 'reported_seq': 11417, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.918925+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:24.918925+0000', 'last_peered': '2024-09-15T21:13:24.918925+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:24.918925+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10687756, 'num_objects': 1305, 'num_object_clones': 0, 'num_object_copies': 5220, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1305, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11352, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1006}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11107", 'reported_seq': 11158, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.933310+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:24.933310+0000', 'last_peered': '2024-09-15T21:13:24.933310+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:24.933310+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10395648, 'num_objects': 1269, 'num_object_clones': 0, 'num_object_copies': 5076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11107, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1047}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.712 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11420", 'reported_seq': 11496, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:22.766091+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:22.766091+0000', 'last_peered': '2024-09-15T21:13:22.766091+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:22.766091+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11072780, 'num_objects': 1352, 'num_object_clones': 0, 'num_object_copies': 5408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11418, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1014}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11411", 'reported_seq': 11486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:22.715735+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:22.715735+0000', 'last_peered': '2024-09-15T21:13:22.715735+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:22.715735+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11056396, 'num_objects': 1350, 'num_object_clones': 0, 'num_object_copies': 5400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1350, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11409, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1124}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11138", 'reported_seq': 11203, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.811988+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:24.811988+0000', 'last_peered': '2024-09-15T21:13:24.811988+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:24.811988+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10758680, 'num_objects': 1314, 'num_object_clones': 0, 'num_object_copies': 5256, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1314, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11135, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1018}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11241", 'reported_seq': 11328, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:25.871363+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:25.871363+0000', 'last_peered': '2024-09-15T21:13:25.871363+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:25.871363+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10041, 'log_dups_size': 0, 'ondisk_log_size': 10041, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10818828, 'num_objects': 1321, 'num_object_clones': 0, 'num_object_copies': 5284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1321, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11239, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1014}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:26.713 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11068", 'reported_seq': 11114, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.836680+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:24.836680+0000', 'last_peered': '2024-09-15T21:13:24.836680+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:24.836680+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1000", 'ondisk_log_start': "27'1000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11131526, 'num_objects': 1359, 'num_object_clones': 0, 'num_object_copies': 5436, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1359, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11067, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -897}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11194", 'reported_seq': 14866, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:25.867696+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:25.867696+0000', 'last_peered': '2024-09-15T21:13:25.867696+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:25.867696+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10705542, 'num_objects': 1307, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 494, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11193, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1794, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 813}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11142", 'reported_seq': 11205, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:24.706602+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:24.706602+0000', 'last_peered': '2024-09-15T21:13:24.706602+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:24.706602+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10564876, 'num_objects': 1290, 'num_object_clones': 0, 'num_object_copies': 5160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1290, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11140, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11274", 'reported_seq': 11363, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:25.867572+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:25.867572+0000', 'last_peered': '2024-09-15T21:13:25.867572+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:25.867572+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11523340, 'num_objects': 1407, 'num_object_clones': 0, 'num_object_copies': 5628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1407, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11272, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -874}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:26.714 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11200", 'reported_seq': 11274, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:22.763536+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:22.763536+0000', 'last_peered': '2024-09-15T21:13:22.763536+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:22.763536+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 11048204, 'num_objects': 1349, 'num_object_clones': 0, 'num_object_copies': 5396, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1349, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11198, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -966}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:26.715 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:27.472 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:27.472 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30648/84648 objects degraded (36.206%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":673,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":21162,"data_bytes":173318690,"bytes_used":2834219008,"bytes_avail":383712837632,"bytes_total":386547056640,"degraded_objects":30648,"degraded_total":84648,"degraded_ratio":0.36206407711936495,"recovering_objects_per_sec":14,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":217},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:27.495 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:27.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:28.132 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:28.133 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:28.155 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-15T21:13:26.166223+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":173318690,"num_objects":21162,"num_object_clones":0,"num_object_copies":84648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163916,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2767792,"kb_used_data":342564,"kb_used_omap":14,"kb_used_meta":2425073,"kb_avail":374719568,"statfs":{"total":386547056640,"available":383712837632,"internally_reserved":0,"allocated":350785536,"data_stored":350093128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2483275314},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,8,7,12,19,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21430272,"num_objects":-2616,"num_object_clones":0,"num_object_copies":-10464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2616,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004520"},"pg_stats":[{"pgid":"1.f","version":"35'11080","reported_seq":11131,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.986868+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:24.986868+0000","last_peered":"2024-09-15T21:13:24.986868+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:24.986868+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10420224,"num_objects":1272,"num_object_clones":0,"num_object_copies":5088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11080,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1004}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11224","reported_seq":16118,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:23.264573+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:23.264573+0000","last_peered":"2024-09-15T21:13:23.264573+0000","last_clean":"2024-09-15T21:13:23.264573+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:23.264573+0000","last_undegraded":"2024-09-15T21:13:23.264573+0000","last_fullsized":"2024-09-15T21:13:23.264573+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10862592,"num_objects":1326,"num_object_clones":0,"num_object_copies":5304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11094","reported_seq":11180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.698479+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:25.698479+0000","last_peered":"2024-09-15T21:13:25.698479+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:25.698479+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10959518,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11093,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11146","reported_seq":11222,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.573715+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:22.573715+0000","last_peered":"2024-09-15T21:13:22.573715+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:22.573715+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10728716,"num_objects":1310,"num_object_clones":0,"num_object_copies":5240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11354","reported_seq":11417,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.918925+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:24.918925+0000","last_peered":"2024-09-15T21:13:24.918925+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:24.918925+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10687756,"num_objects":1305,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11352,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1006}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11107","reported_seq":11158,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.933310+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:24.933310+0000","last_peered":"2024-09-15T21:13:24.933310+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:24.933310+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10395648,"num_objects":1269,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11107,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1047}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11420","reported_seq":11496,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.766091+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:22.766091+0000","last_peered":"2024-09-15T21:13:22.766091+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:22.766091+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11072780,"num_objects":1352,"num_object_clones":0,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11418,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1014}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11411","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.715735+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:22.715735+0000","last_peered":"2024-09-15T21:13:22.715735+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:22.715735+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11056396,"num_objects":1350,"num_object_clones":0,"num_object_copies":5400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11409,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1124}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11138","reported_seq":11203,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.811988+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:24.811988+0000","last_peered":"2024-09-15T21:13:24.811988+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:24.811988+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10758680,"num_objects":1314,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11135,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1018}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11241","reported_seq":11328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.871363+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:25.871363+0000","last_peered":"2024-09-15T21:13:25.871363+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:25.871363+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10818828,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11068","reported_seq":15737,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:24.947293+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:24.947293+0000","last_peered":"2024-09-15T21:13:24.947293+0000","last_clean":"2024-09-15T21:13:24.947293+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:24.947293+0000","last_undegraded":"2024-09-15T21:13:24.947293+0000","last_fullsized":"2024-09-15T21:13:24.947293+0000","mapping_epoch":32,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10584064,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5179,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11068","reported_seq":11114,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.836680+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:24.836680+0000","last_peered":"2024-09-15T21:13:24.836680+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:24.836680+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11131526,"num_objects":1359,"num_object_clones":0,"num_object_copies":5436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-897}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11194","reported_seq":14866,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867696+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:25.867696+0000","last_peered":"2024-09-15T21:13:25.867696+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:25.867696+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11142","reported_seq":11205,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:24.706602+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:24.706602+0000","last_peered":"2024-09-15T21:13:24.706602+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:24.706602+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10564876,"num_objects":1290,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11140,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11274","reported_seq":11363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867572+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:25.867572+0000","last_peered":"2024-09-15T21:13:25.867572+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:25.867572+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523340,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11200","reported_seq":11274,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:22.763536+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:22.763536+0000","last_peered":"2024-09-15T21:13:22.763536+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:22.763536+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11048204,"num_objects":1349,"num_object_clones":0,"num_object_copies":5396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11198,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-966}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":173318690,"num_objects":21162,"num_object_clones":0,"num_object_copies":84648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21162,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163916,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":347394048,"data_stored":347394048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160861,"ondisk_log_size":160861,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":716656,"kb_used_data":85188,"kb_used_omap":4,"kb_used_meta":631419,"kb_avail":93655184,"statfs":{"total":96636764160,"available":95902908416,"internally_reserved":0,"allocated":87232512,"data_stored":87059410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":646573776},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719728,"kb_used_data":84640,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93652112,"statfs":{"total":96636764160,"available":95899762688,"internally_reserved":0,"allocated":86671360,"data_stored":86498258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051697,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591436,"kb_used_data":86356,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93780404,"statfs":{"total":96636764160,"available":96031133696,"internally_reserved":0,"allocated":88428544,"data_stored":88255442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247935,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":739972,"kb_used_data":86380,"kb_used_omap":4,"kb_used_meta":653563,"kb_avail":93631868,"statfs":{"total":96636764160,"available":95879032832,"internally_reserved":0,"allocated":88453120,"data_stored":88280018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":669249230},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,9,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":88006656,"data_stored":88006656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":87588864,"data_stored":87588864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85528576,"data_stored":85528576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86269952,"data_stored":86269952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:31.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:31.757 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:31.758 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:31.794 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-15T21:13:30.167752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":167404066,"num_objects":20440,"num_object_clones":0,"num_object_copies":81760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164638,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788024,"kb_used_data":330936,"kb_used_omap":14,"kb_used_meta":2457009,"kb_avail":374699336,"statfs":{"total":386547056640,"available":383692120064,"internally_reserved":0,"allocated":338878464,"data_stored":338186056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2515977778},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,7,9,23,8,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19718144,"num_objects":-2407,"num_object_clones":0,"num_object_copies":-9628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004612"},"pg_stats":[{"pgid":"1.f","version":"35'11143","reported_seq":11195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.812184+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:29.812184+0000","last_peered":"2024-09-15T21:13:29.812184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:29.812184+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9904128,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11296","reported_seq":16190,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:28.430353+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:28.430353+0000","last_peered":"2024-09-15T21:13:28.430353+0000","last_clean":"2024-09-15T21:13:28.430353+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:28.430353+0000","last_undegraded":"2024-09-15T21:13:28.430353+0000","last_fullsized":"2024-09-15T21:13:28.430353+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10272768,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11094","reported_seq":11180,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.698479+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:25.698479+0000","last_peered":"2024-09-15T21:13:25.698479+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:25.698479+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1000","ondisk_log_start":"27'1000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10959518,"num_objects":1339,"num_object_clones":0,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11093,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-919}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11215","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.838098+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:27.838098+0000","last_peered":"2024-09-15T21:13:27.838098+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:27.838098+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163468,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11423","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.852098+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:29.852098+0000","last_peered":"2024-09-15T21:13:29.852098+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:29.852098+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10122508,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11169","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.910147+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:29.910147+0000","last_peered":"2024-09-15T21:13:29.910147+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:29.910147+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9887744,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11479","reported_seq":11555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.731135+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:27.731135+0000","last_peered":"2024-09-15T21:13:27.731135+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:27.731135+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10589452,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11463","reported_seq":11538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.685283+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:27.685283+0000","last_peered":"2024-09-15T21:13:27.685283+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:27.685283+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10630412,"num_objects":1298,"num_object_clones":0,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11194","reported_seq":11260,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.968930+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:29.968930+0000","last_peered":"2024-09-15T21:13:29.968930+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:29.968930+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10299928,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11191,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11241","reported_seq":11328,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.871363+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:25.871363+0000","last_peered":"2024-09-15T21:13:25.871363+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:25.871363+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10818828,"num_objects":1321,"num_object_clones":0,"num_object_copies":5284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11239,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1014}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11116","reported_seq":15786,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:29.956483+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:29.956483+0000","last_peered":"2024-09-15T21:13:29.956483+0000","last_clean":"2024-09-15T21:13:29.956483+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:29.956483+0000","last_undegraded":"2024-09-15T21:13:29.956483+0000","last_fullsized":"2024-09-15T21:13:29.956483+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10190848,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11120","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.919888+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:29.919888+0000","last_peered":"2024-09-15T21:13:29.919888+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:29.919888+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-949}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11194","reported_seq":14866,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867696+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:25.867696+0000","last_peered":"2024-09-15T21:13:25.867696+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:25.867696+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":494,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11193,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1794,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":813}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11205","reported_seq":11268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.858007+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:29.858007+0000","last_peered":"2024-09-15T21:13:29.858007+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:29.858007+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048780,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11274","reported_seq":11363,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:25.867572+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:25.867572+0000","last_peered":"2024-09-15T21:13:25.867572+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:25.867572+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11523340,"num_objects":1407,"num_object_clones":0,"num_object_copies":5628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11272,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-874}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11257","reported_seq":11332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.664824+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:27.664824+0000","last_peered":"2024-09-15T21:13:27.664824+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:27.664824+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581260,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11255,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":167404066,"num_objects":20440,"num_object_clones":0,"num_object_copies":81760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20440,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164638,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":335372288,"data_stored":335372288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160883,"ondisk_log_size":160883,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735000,"kb_used_data":81352,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636840,"statfs":{"total":96636764160,"available":95884124160,"internally_reserved":0,"allocated":83304448,"data_stored":83131346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719728,"kb_used_data":84640,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93652112,"statfs":{"total":96636764160,"available":95899762688,"internally_reserved":0,"allocated":86671360,"data_stored":86498258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591692,"kb_used_data":82492,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93780148,"statfs":{"total":96636764160,"available":96030871552,"internally_reserved":0,"allocated":84471808,"data_stored":84298706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741604,"kb_used_data":82452,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93630236,"statfs":{"total":96636764160,"available":95877361664,"internally_reserved":0,"allocated":84430848,"data_stored":84257746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84033536,"data_stored":84033536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83533824,"data_stored":83533824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85528576,"data_stored":85528576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82276352,"data_stored":82276352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:31.795 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:32.472 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:32.473 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:32.496 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-15T21:13:32.168609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165290530,"num_objects":20182,"num_object_clones":0,"num_object_copies":80728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164896,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1877,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788984,"kb_used_data":326324,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374698376,"statfs":{"total":386547056640,"available":383691137024,"internally_reserved":0,"allocated":334155776,"data_stored":333463368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,5,9,9,24,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004778"},"pg_stats":[{"pgid":"1.f","version":"35'11143","reported_seq":11195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.812184+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:29.812184+0000","last_peered":"2024-09-15T21:13:29.812184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:29.812184+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9904128,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11296","reported_seq":16190,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:28.430353+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:28.430353+0000","last_peered":"2024-09-15T21:13:28.430353+0000","last_clean":"2024-09-15T21:13:28.430353+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:28.430353+0000","last_undegraded":"2024-09-15T21:13:28.430353+0000","last_fullsized":"2024-09-15T21:13:28.430353+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10272768,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11156","reported_seq":11242,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.825582+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:30.825582+0000","last_peered":"2024-09-15T21:13:30.825582+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:30.825582+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10451614,"num_objects":1277,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11155,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11215","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.838098+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:27.838098+0000","last_peered":"2024-09-15T21:13:27.838098+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:27.838098+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163468,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11423","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.852098+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:29.852098+0000","last_peered":"2024-09-15T21:13:29.852098+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:29.852098+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10122508,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11169","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.910147+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:29.910147+0000","last_peered":"2024-09-15T21:13:29.910147+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:29.910147+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9887744,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11479","reported_seq":11555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.731135+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:27.731135+0000","last_peered":"2024-09-15T21:13:27.731135+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:27.731135+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10589452,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11463","reported_seq":11538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.685283+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:27.685283+0000","last_peered":"2024-09-15T21:13:27.685283+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:27.685283+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10630412,"num_objects":1298,"num_object_clones":0,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11194","reported_seq":11260,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.968930+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:29.968930+0000","last_peered":"2024-09-15T21:13:29.968930+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:29.968930+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10299928,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11191,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11308","reported_seq":11395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.877687+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:30.877687+0000","last_peered":"2024-09-15T21:13:30.877687+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:30.877687+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10269964,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11116","reported_seq":15786,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:29.956483+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:29.956483+0000","last_peered":"2024-09-15T21:13:29.956483+0000","last_clean":"2024-09-15T21:13:29.956483+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:29.956483+0000","last_undegraded":"2024-09-15T21:13:29.956483+0000","last_fullsized":"2024-09-15T21:13:29.956483+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10190848,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11120","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.919888+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:29.919888+0000","last_peered":"2024-09-15T21:13:29.919888+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:29.919888+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-949}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11249","reported_seq":15046,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:31.012257+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:31.012257+0000","last_peered":"2024-09-15T21:13:31.012257+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:31.012257+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10254982,"num_objects":1252,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11205","reported_seq":11268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.858007+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:29.858007+0000","last_peered":"2024-09-15T21:13:29.858007+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:29.858007+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048780,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11348","reported_seq":11437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.967442+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:30.967442+0000","last_peered":"2024-09-15T21:13:30.967442+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:30.967442+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10917132,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-948}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11257","reported_seq":11332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.664824+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:27.664824+0000","last_peered":"2024-09-15T21:13:27.664824+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:27.664824+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581260,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11255,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165290530,"num_objects":20182,"num_object_clones":0,"num_object_copies":80728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164896,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1877,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331317248,"data_stored":331317248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735000,"kb_used_data":81352,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636840,"statfs":{"total":96636764160,"available":95884124160,"internally_reserved":0,"allocated":83304448,"data_stored":83131346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720688,"kb_used_data":80028,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93651152,"statfs":{"total":96636764160,"available":95898779648,"internally_reserved":0,"allocated":81948672,"data_stored":81775570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,3,1,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591692,"kb_used_data":82492,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93780148,"statfs":{"total":96636764160,"available":96030871552,"internally_reserved":0,"allocated":84471808,"data_stored":84298706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741604,"kb_used_data":82452,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93630236,"statfs":{"total":96636764160,"available":95877361664,"internally_reserved":0,"allocated":84430848,"data_stored":84257746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84033536,"data_stored":84033536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83533824,"data_stored":83533824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81473536,"data_stored":81473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82276352,"data_stored":82276352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:32.497 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:32.497 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11143", 'reported_seq': 11195, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.812184+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:29.812184+0000', 'last_peered': '2024-09-15T21:13:29.812184+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:29.812184+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9904128, 'num_objects': 1209, 'num_object_clones': 0, 'num_object_copies': 4836, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1209, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11143, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1067}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.497 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:32.497 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11156", 'reported_seq': 11242, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:30.825582+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:30.825582+0000', 'last_peered': '2024-09-15T21:13:30.825582+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:30.825582+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10451614, 'num_objects': 1277, 'num_object_clones': 0, 'num_object_copies': 5108, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1277, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11155, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -981}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:32.497 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11215", 'reported_seq': 11291, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:27.838098+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:27.838098+0000', 'last_peered': '2024-09-15T21:13:27.838098+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:27.838098+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10163468, 'num_objects': 1241, 'num_object_clones': 0, 'num_object_copies': 4964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1241, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11213, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1093}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11423", 'reported_seq': 11486, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.852098+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:29.852098+0000', 'last_peered': '2024-09-15T21:13:29.852098+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:29.852098+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10122508, 'num_objects': 1236, 'num_object_clones': 0, 'num_object_copies': 4944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1236, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11421, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1075}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11169", 'reported_seq': 11220, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.910147+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:29.910147+0000', 'last_peered': '2024-09-15T21:13:29.910147+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:29.910147+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9887744, 'num_objects': 1207, 'num_object_clones': 0, 'num_object_copies': 4828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11169, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1109}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11479", 'reported_seq': 11555, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:27.731135+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:27.731135+0000', 'last_peered': '2024-09-15T21:13:27.731135+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:27.731135+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10589452, 'num_objects': 1293, 'num_object_clones': 0, 'num_object_copies': 5172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11477, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1073}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11463", 'reported_seq': 11538, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:27.685283+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:27.685283+0000', 'last_peered': '2024-09-15T21:13:27.685283+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:27.685283+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10630412, 'num_objects': 1298, 'num_object_clones': 0, 'num_object_copies': 5192, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1298, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11461, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1176}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:32.498 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11194", 'reported_seq': 11260, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.968930+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:29.968930+0000', 'last_peered': '2024-09-15T21:13:29.968930+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:29.968930+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10094, 'log_dups_size': 0, 'ondisk_log_size': 10094, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10299928, 'num_objects': 1258, 'num_object_clones': 0, 'num_object_copies': 5032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1258, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11191, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1074}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11308", 'reported_seq': 11395, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:30.877687+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:30.877687+0000', 'last_peered': '2024-09-15T21:13:30.877687+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:30.877687+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10269964, 'num_objects': 1254, 'num_object_clones': 0, 'num_object_copies': 5016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11306, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1081}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11120", 'reported_seq': 11166, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.919888+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:29.919888+0000', 'last_peered': '2024-09-15T21:13:29.919888+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:29.919888+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10705542, 'num_objects': 1307, 'num_object_clones': 0, 'num_object_copies': 5228, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1307, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11119, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -949}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11249", 'reported_seq': 15046, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:31.012257+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:31.012257+0000', 'last_peered': '2024-09-15T21:13:31.012257+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:31.012257+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10254982, 'num_objects': 1252, 'num_object_clones': 0, 'num_object_copies': 5008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 432, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1252, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11248, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1856, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 820}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11205", 'reported_seq': 11268, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:29.858007+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:29.858007+0000', 'last_peered': '2024-09-15T21:13:29.858007+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:29.858007+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10048780, 'num_objects': 1227, 'num_object_clones': 0, 'num_object_copies': 4908, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1227, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11203, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:32.499 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11348", 'reported_seq': 11437, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:30.967442+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:30.967442+0000', 'last_peered': '2024-09-15T21:13:30.967442+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:30.967442+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10917132, 'num_objects': 1333, 'num_object_clones': 0, 'num_object_copies': 5332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1333, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11346, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -948}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:32.500 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:32.500 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11257", 'reported_seq': 11332, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:27.664824+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:27.664824+0000', 'last_peered': '2024-09-15T21:13:27.664824+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:27.664824+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10581260, 'num_objects': 1292, 'num_object_clones': 0, 'num_object_copies': 5168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11255, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1023}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:32.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:33.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:33.240 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30586/80728 objects degraded (37.888%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":678,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20182,"data_bytes":165290530,"bytes_used":2855919616,"bytes_avail":383691137024,"bytes_total":386547056640,"degraded_objects":30586,"degraded_total":80728,"degraded_ratio":0.3788772173223664,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:33.271 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:33.271 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:33.906 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:33.907 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:33.940 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-15T21:13:32.168609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":165290530,"num_objects":20182,"num_object_clones":0,"num_object_copies":80728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164896,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1877,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2788984,"kb_used_data":326324,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374698376,"statfs":{"total":386547056640,"available":383691137024,"internally_reserved":0,"allocated":334155776,"data_stored":333463368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,1,5,9,9,24,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18128896,"num_objects":-2213,"num_object_clones":0,"num_object_copies":-8852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-185,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2213,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":185,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004778"},"pg_stats":[{"pgid":"1.f","version":"35'11143","reported_seq":11195,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.812184+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:29.812184+0000","last_peered":"2024-09-15T21:13:29.812184+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:29.812184+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9904128,"num_objects":1209,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11143,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1067}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11296","reported_seq":16190,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:28.430353+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:28.430353+0000","last_peered":"2024-09-15T21:13:28.430353+0000","last_clean":"2024-09-15T21:13:28.430353+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:28.430353+0000","last_undegraded":"2024-09-15T21:13:28.430353+0000","last_fullsized":"2024-09-15T21:13:28.430353+0000","mapping_epoch":34,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10272768,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1962,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11156","reported_seq":11242,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.825582+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:30.825582+0000","last_peered":"2024-09-15T21:13:30.825582+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:30.825582+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10451614,"num_objects":1277,"num_object_clones":0,"num_object_copies":5108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11155,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-981}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11215","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.838098+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:27.838098+0000","last_peered":"2024-09-15T21:13:27.838098+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:27.838098+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10163468,"num_objects":1241,"num_object_clones":0,"num_object_copies":4964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11213,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1093}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11423","reported_seq":11486,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.852098+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:29.852098+0000","last_peered":"2024-09-15T21:13:29.852098+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:29.852098+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10122508,"num_objects":1236,"num_object_clones":0,"num_object_copies":4944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1075}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11169","reported_seq":11220,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.910147+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:29.910147+0000","last_peered":"2024-09-15T21:13:29.910147+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:29.910147+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9887744,"num_objects":1207,"num_object_clones":0,"num_object_copies":4828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11169,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1109}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11479","reported_seq":11555,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.731135+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:27.731135+0000","last_peered":"2024-09-15T21:13:27.731135+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:27.731135+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10589452,"num_objects":1293,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11463","reported_seq":11538,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.685283+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:27.685283+0000","last_peered":"2024-09-15T21:13:27.685283+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:27.685283+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10630412,"num_objects":1298,"num_object_clones":0,"num_object_copies":5192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1176}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11194","reported_seq":11260,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.968930+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:29.968930+0000","last_peered":"2024-09-15T21:13:29.968930+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:29.968930+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10299928,"num_objects":1258,"num_object_clones":0,"num_object_copies":5032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11191,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1074}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11308","reported_seq":11395,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.877687+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:30.877687+0000","last_peered":"2024-09-15T21:13:30.877687+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:30.877687+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10269964,"num_objects":1254,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11116","reported_seq":15786,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:29.956483+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:29.956483+0000","last_peered":"2024-09-15T21:13:29.956483+0000","last_clean":"2024-09-15T21:13:29.956483+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:29.956483+0000","last_undegraded":"2024-09-15T21:13:29.956483+0000","last_fullsized":"2024-09-15T21:13:29.956483+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10190848,"num_objects":1244,"num_object_clones":0,"num_object_copies":4976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5227,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11120","reported_seq":11166,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.919888+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:29.919888+0000","last_peered":"2024-09-15T21:13:29.919888+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:29.919888+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10705542,"num_objects":1307,"num_object_clones":0,"num_object_copies":5228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-949}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11249","reported_seq":15046,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:31.012257+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:31.012257+0000","last_peered":"2024-09-15T21:13:31.012257+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:31.012257+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10254982,"num_objects":1252,"num_object_clones":0,"num_object_copies":5008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":820}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11205","reported_seq":11268,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:29.858007+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:29.858007+0000","last_peered":"2024-09-15T21:13:29.858007+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:29.858007+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10048780,"num_objects":1227,"num_object_clones":0,"num_object_copies":4908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1052}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11348","reported_seq":11437,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:30.967442+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:30.967442+0000","last_peered":"2024-09-15T21:13:30.967442+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:30.967442+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10917132,"num_objects":1333,"num_object_clones":0,"num_object_copies":5332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-948}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11257","reported_seq":11332,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:27.664824+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:27.664824+0000","last_peered":"2024-09-15T21:13:27.664824+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:27.664824+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10581260,"num_objects":1292,"num_object_clones":0,"num_object_copies":5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11255,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1023}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":165290530,"num_objects":20182,"num_object_clones":0,"num_object_copies":80728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30586,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20182,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164896,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1877,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":331317248,"data_stored":331317248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":735000,"kb_used_data":81352,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93636840,"statfs":{"total":96636764160,"available":95884124160,"internally_reserved":0,"allocated":83304448,"data_stored":83131346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,0,9,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720688,"kb_used_data":80028,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93651152,"statfs":{"total":96636764160,"available":95898779648,"internally_reserved":0,"allocated":81948672,"data_stored":81775570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,1,3,1,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051698,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591692,"kb_used_data":82492,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93780148,"statfs":{"total":96636764160,"available":96030871552,"internally_reserved":0,"allocated":84471808,"data_stored":84298706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,5,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247936,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741604,"kb_used_data":82452,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93630236,"statfs":{"total":96636764160,"available":95877361664,"internally_reserved":0,"allocated":84430848,"data_stored":84257746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":84033536,"data_stored":84033536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83533824,"data_stored":83533824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81473536,"data_stored":81473536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82276352,"data_stored":82276352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:36.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:37.584 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:37.584 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:37.607 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-15T21:13:36.169788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157114914,"num_objects":19184,"num_object_clones":0,"num_object_copies":76736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165894,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769196,"kb_used_data":310700,"kb_used_omap":14,"kb_used_meta":2458417,"kb_avail":374718164,"statfs":{"total":386547056640,"available":383711399936,"internally_reserved":0,"allocated":318156800,"data_stored":317464392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2517419570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,5,1,9,12,9,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21626880,"num_objects":-2640,"num_object_clones":0,"num_object_copies":-10560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2640,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004274"},"pg_stats":[{"pgid":"1.f","version":"35'11199","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.909000+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:34.909000+0000","last_peered":"2024-09-15T21:13:34.909000+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:34.909000+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445376,"num_objects":1153,"num_object_clones":0,"num_object_copies":4612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11365","reported_seq":16259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:33.524942+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:33.524942+0000","last_peered":"2024-09-15T21:13:33.524942+0000","last_clean":"2024-09-15T21:13:33.524942+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:33.524942+0000","last_undegraded":"2024-09-15T21:13:33.524942+0000","last_fullsized":"2024-09-15T21:13:33.524942+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9707520,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11209","reported_seq":11295,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.837829+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:35.837829+0000","last_peered":"2024-09-15T21:13:35.837829+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:35.837829+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017438,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11208,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11265","reported_seq":11341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:32.837349+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:32.837349+0000","last_peered":"2024-09-15T21:13:32.837349+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:32.837349+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9753868,"num_objects":1191,"num_object_clones":0,"num_object_copies":4764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11263,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11479","reported_seq":11542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.010609+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:35.010609+0000","last_peered":"2024-09-15T21:13:35.010609+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:35.010609+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663756,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11235","reported_seq":11286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.949643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:34.949643+0000","last_peered":"2024-09-15T21:13:34.949643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:34.949643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9347072,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11543","reported_seq":11618,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:32.737400+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:32.737400+0000","last_peered":"2024-09-15T21:13:32.737400+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:32.737400+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10065164,"num_objects":1229,"num_object_clones":0,"num_object_copies":4916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1137}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11545","reported_seq":11619,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:32.885135+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:32.885135+0000","last_peered":"2024-09-15T21:13:32.885135+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:32.885135+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9958668,"num_objects":1216,"num_object_clones":0,"num_object_copies":4864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11543,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1258}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11256","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.728370+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:34.728370+0000","last_peered":"2024-09-15T21:13:34.728370+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:34.728370+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9792024,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11253,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11375","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.108346+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:36.108346+0000","last_peered":"2024-09-15T21:13:36.108346+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:36.108346+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9721100,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11164","reported_seq":15834,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:34.858214+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:34.858214+0000","last_peered":"2024-09-15T21:13:34.858214+0000","last_clean":"2024-09-15T21:13:34.858214+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:34.858214+0000","last_undegraded":"2024-09-15T21:13:34.858214+0000","last_fullsized":"2024-09-15T21:13:34.858214+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9797632,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11187","reported_seq":11233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.822835+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:34.822835+0000","last_peered":"2024-09-15T21:13:34.822835+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:34.822835+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10156678,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11313","reported_seq":15232,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.063199+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:36.063199+0000","last_peered":"2024-09-15T21:13:36.063199+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:36.063199+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9730694,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11260","reported_seq":11323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.053177+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:35.053177+0000","last_peered":"2024-09-15T21:13:35.053177+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:35.053177+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9598220,"num_objects":1172,"num_object_clones":0,"num_object_copies":4688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11425","reported_seq":11514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.990075+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:35.990075+0000","last_peered":"2024-09-15T21:13:35.990075+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:35.990075+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286348,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11423,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11319","reported_seq":11394,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:32.966838+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:32.966838+0000","last_peered":"2024-09-15T21:13:32.966838+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:32.966838+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10073356,"num_objects":1230,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11317,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1085}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":157114914,"num_objects":19184,"num_object_clones":0,"num_object_copies":76736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165894,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315142144,"data_stored":315142144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160839,"ondisk_log_size":160839,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731044,"kb_used_data":77396,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93640796,"statfs":{"total":96636764160,"available":95888175104,"internally_reserved":0,"allocated":79253504,"data_stored":79080402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,0,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721060,"kb_used_data":76244,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93650780,"statfs":{"total":96636764160,"available":95898398720,"internally_reserved":0,"allocated":78073856,"data_stored":77900754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575264,"kb_used_data":78544,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93796576,"statfs":{"total":96636764160,"available":96047693824,"internally_reserved":0,"allocated":80429056,"data_stored":80255954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741828,"kb_used_data":78516,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93630012,"statfs":{"total":96636764160,"available":95877132288,"internally_reserved":0,"allocated":80400384,"data_stored":80227282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79970304,"data_stored":79970304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79417344,"data_stored":79417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77463552,"data_stored":77463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78290944,"data_stored":78290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:37.608 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:38.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:38.256 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:38.278 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-15T21:13:38.170495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":155181602,"num_objects":18948,"num_object_clones":0,"num_object_copies":75792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166130,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769364,"kb_used_data":306688,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374717996,"statfs":{"total":386547056640,"available":383711227904,"internally_reserved":0,"allocated":314048512,"data_stored":313356104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,5,3,15,12,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18137088,"num_objects":-2214,"num_object_clones":0,"num_object_copies":-8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"35'11199","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.909000+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:34.909000+0000","last_peered":"2024-09-15T21:13:34.909000+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:34.909000+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445376,"num_objects":1153,"num_object_clones":0,"num_object_copies":4612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11365","reported_seq":16259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:33.524942+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:33.524942+0000","last_peered":"2024-09-15T21:13:33.524942+0000","last_clean":"2024-09-15T21:13:33.524942+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:33.524942+0000","last_undegraded":"2024-09-15T21:13:33.524942+0000","last_fullsized":"2024-09-15T21:13:33.524942+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9707520,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11209","reported_seq":11295,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.837829+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:35.837829+0000","last_peered":"2024-09-15T21:13:35.837829+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:35.837829+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017438,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11208,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11327","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.978559+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:37.978559+0000","last_peered":"2024-09-15T21:13:37.978559+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:37.978559+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11479","reported_seq":11542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.010609+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:35.010609+0000","last_peered":"2024-09-15T21:13:35.010609+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:35.010609+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663756,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11235","reported_seq":11286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.949643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:34.949643+0000","last_peered":"2024-09-15T21:13:34.949643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:34.949643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9347072,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11604","reported_seq":11681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.944736+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:37.944736+0000","last_peered":"2024-09-15T21:13:37.944736+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:37.944736+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1198}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11596","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.837751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:37.837751+0000","last_peered":"2024-09-15T21:13:37.837751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:37.837751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540876,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11256","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.728370+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:34.728370+0000","last_peered":"2024-09-15T21:13:34.728370+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:34.728370+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9792024,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11253,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11375","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.108346+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:36.108346+0000","last_peered":"2024-09-15T21:13:36.108346+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:36.108346+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9721100,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11164","reported_seq":15834,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:34.858214+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:34.858214+0000","last_peered":"2024-09-15T21:13:34.858214+0000","last_clean":"2024-09-15T21:13:34.858214+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:34.858214+0000","last_undegraded":"2024-09-15T21:13:34.858214+0000","last_fullsized":"2024-09-15T21:13:34.858214+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9797632,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11187","reported_seq":11233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.822835+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:34.822835+0000","last_peered":"2024-09-15T21:13:34.822835+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:34.822835+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10156678,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11313","reported_seq":15232,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.063199+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:36.063199+0000","last_peered":"2024-09-15T21:13:36.063199+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:36.063199+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9730694,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11260","reported_seq":11323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.053177+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:35.053177+0000","last_peered":"2024-09-15T21:13:35.053177+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:35.053177+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9598220,"num_objects":1172,"num_object_clones":0,"num_object_copies":4688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11425","reported_seq":11514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.990075+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:35.990075+0000","last_peered":"2024-09-15T21:13:35.990075+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:35.990075+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286348,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11423,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11381","reported_seq":11454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.969615+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:37.969615+0000","last_peered":"2024-09-15T21:13:37.969615+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:37.969615+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":155181602,"num_objects":18948,"num_object_clones":0,"num_object_copies":75792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166130,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311095296,"data_stored":311095296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731044,"kb_used_data":77396,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93640796,"statfs":{"total":96636764160,"available":95888175104,"internally_reserved":0,"allocated":79253504,"data_stored":79080402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,0,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721060,"kb_used_data":76244,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93650780,"statfs":{"total":96636764160,"available":95898398720,"internally_reserved":0,"allocated":78073856,"data_stored":77900754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575264,"kb_used_data":78544,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93796576,"statfs":{"total":96636764160,"available":96047693824,"internally_reserved":0,"allocated":80429056,"data_stored":80255954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741996,"kb_used_data":74504,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93629844,"statfs":{"total":96636764160,"available":95876960256,"internally_reserved":0,"allocated":76292096,"data_stored":76118994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,6,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75923456,"data_stored":75923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79417344,"data_stored":79417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77463552,"data_stored":77463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78290944,"data_stored":78290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11199", 'reported_seq': 11251, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:34.909000+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:34.909000+0000', 'last_peered': '2024-09-15T21:13:34.909000+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:34.909000+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9445376, 'num_objects': 1153, 'num_object_clones': 0, 'num_object_copies': 4612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11199, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11209", 'reported_seq': 11295, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:35.837829+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:35.837829+0000', 'last_peered': '2024-09-15T21:13:35.837829+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:35.837829+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10009, 'log_dups_size': 0, 'ondisk_log_size': 10009, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10017438, 'num_objects': 1224, 'num_object_clones': 0, 'num_object_copies': 4896, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1224, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11208, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1034}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11327", 'reported_seq': 11401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.978559+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:37.978559+0000', 'last_peered': '2024-09-15T21:13:37.978559+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:37.978559+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9245964, 'num_objects': 1129, 'num_object_clones': 0, 'num_object_copies': 4516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11325, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:38.280 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11479", 'reported_seq': 11542, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:35.010609+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:35.010609+0000', 'last_peered': '2024-09-15T21:13:35.010609+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:35.010609+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9663756, 'num_objects': 1180, 'num_object_clones': 0, 'num_object_copies': 4720, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1180, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11477, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1131}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11235", 'reported_seq': 11286, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:34.949643+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:34.949643+0000', 'last_peered': '2024-09-15T21:13:34.949643+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:34.949643+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9347072, 'num_objects': 1141, 'num_object_clones': 0, 'num_object_copies': 4564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11235, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1175}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11604", 'reported_seq': 11681, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.944736+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:37.944736+0000', 'last_peered': '2024-09-15T21:13:37.944736+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:37.944736+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11602, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1198}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:38.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11596", 'reported_seq': 11672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.837751+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:37.837751+0000', 'last_peered': '2024-09-15T21:13:37.837751+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:37.837751+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9540876, 'num_objects': 1165, 'num_object_clones': 0, 'num_object_copies': 4660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11594, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11256", 'reported_seq': 11320, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:34.728370+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:34.728370+0000', 'last_peered': '2024-09-15T21:13:34.728370+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:34.728370+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9792024, 'num_objects': 1196, 'num_object_clones': 0, 'num_object_copies': 4784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11253, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1136}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:38.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11375", 'reported_seq': 11462, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:36.108346+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:36.108346+0000', 'last_peered': '2024-09-15T21:13:36.108346+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:36.108346+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9721100, 'num_objects': 1187, 'num_object_clones': 0, 'num_object_copies': 4748, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1187, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11373, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1148}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11187", 'reported_seq': 11233, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:34.822835+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:34.822835+0000', 'last_peered': '2024-09-15T21:13:34.822835+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:34.822835+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1100", 'ondisk_log_start': "27'1100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10156678, 'num_objects': 1240, 'num_object_clones': 0, 'num_object_copies': 4960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1240, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11186, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1016}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11313", 'reported_seq': 15232, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:36.063199+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:36.063199+0000', 'last_peered': '2024-09-15T21:13:36.063199+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:36.063199+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9730694, 'num_objects': 1188, 'num_object_clones': 0, 'num_object_copies': 4752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 371, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1188, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11312, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1917, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 817}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11260", 'reported_seq': 11323, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:35.053177+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:35.053177+0000', 'last_peered': '2024-09-15T21:13:35.053177+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:35.053177+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9598220, 'num_objects': 1172, 'num_object_clones': 0, 'num_object_copies': 4688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11258, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1107}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:38.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11425", 'reported_seq': 11514, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:35.990075+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:35.990075+0000', 'last_peered': '2024-09-15T21:13:35.990075+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:35.990075+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 10286348, 'num_objects': 1256, 'num_object_clones': 0, 'num_object_copies': 5024, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1256, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11423, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:38.284 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:38.284 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11381", 'reported_seq': 11454, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.969615+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:37.969615+0000', 'last_peered': '2024-09-15T21:13:37.969615+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:37.969615+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11379, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:38.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:39.035 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:39.036 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30525/75792 objects degraded (40.275%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":684,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18948,"data_bytes":155181602,"bytes_used":2835828736,"bytes_avail":383711227904,"bytes_total":386547056640,"degraded_objects":30525,"degraded_total":75792,"degraded_ratio":0.40274699176694112,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:39.057 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:39.057 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:39.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:39.705 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:39.729 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-15T21:13:38.170495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":155181602,"num_objects":18948,"num_object_clones":0,"num_object_copies":75792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166130,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2769364,"kb_used_data":306688,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374717996,"statfs":{"total":386547056640,"available":383711227904,"internally_reserved":0,"allocated":314048512,"data_stored":313356104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,5,3,15,12,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18137088,"num_objects":-2214,"num_object_clones":0,"num_object_copies":-8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2214,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004271"},"pg_stats":[{"pgid":"1.f","version":"35'11199","reported_seq":11251,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.909000+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:34.909000+0000","last_peered":"2024-09-15T21:13:34.909000+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:34.909000+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9445376,"num_objects":1153,"num_object_clones":0,"num_object_copies":4612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11365","reported_seq":16259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:33.524942+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:33.524942+0000","last_peered":"2024-09-15T21:13:33.524942+0000","last_clean":"2024-09-15T21:13:33.524942+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:33.524942+0000","last_undegraded":"2024-09-15T21:13:33.524942+0000","last_fullsized":"2024-09-15T21:13:33.524942+0000","mapping_epoch":34,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9707520,"num_objects":1185,"num_object_clones":0,"num_object_copies":4740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2031,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11209","reported_seq":11295,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.837829+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:35.837829+0000","last_peered":"2024-09-15T21:13:35.837829+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:35.837829+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10017438,"num_objects":1224,"num_object_clones":0,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1224,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11208,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1034}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11327","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.978559+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:37.978559+0000","last_peered":"2024-09-15T21:13:37.978559+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:37.978559+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11479","reported_seq":11542,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.010609+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:35.010609+0000","last_peered":"2024-09-15T21:13:35.010609+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:35.010609+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9663756,"num_objects":1180,"num_object_clones":0,"num_object_copies":4720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1131}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11235","reported_seq":11286,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.949643+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:34.949643+0000","last_peered":"2024-09-15T21:13:34.949643+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:34.949643+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9347072,"num_objects":1141,"num_object_clones":0,"num_object_copies":4564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1175}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11604","reported_seq":11681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.944736+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:37.944736+0000","last_peered":"2024-09-15T21:13:37.944736+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:37.944736+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1198}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11596","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.837751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:37.837751+0000","last_peered":"2024-09-15T21:13:37.837751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:37.837751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540876,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11256","reported_seq":11320,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.728370+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:34.728370+0000","last_peered":"2024-09-15T21:13:34.728370+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:34.728370+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9792024,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11253,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1136}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11375","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.108346+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:36.108346+0000","last_peered":"2024-09-15T21:13:36.108346+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:36.108346+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9721100,"num_objects":1187,"num_object_clones":0,"num_object_copies":4748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1148}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11164","reported_seq":15834,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:34.858214+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:34.858214+0000","last_peered":"2024-09-15T21:13:34.858214+0000","last_clean":"2024-09-15T21:13:34.858214+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:34.858214+0000","last_undegraded":"2024-09-15T21:13:34.858214+0000","last_fullsized":"2024-09-15T21:13:34.858214+0000","mapping_epoch":32,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9797632,"num_objects":1196,"num_object_clones":0,"num_object_copies":4784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5275,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11187","reported_seq":11233,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:34.822835+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:34.822835+0000","last_peered":"2024-09-15T21:13:34.822835+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:34.822835+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1100","ondisk_log_start":"27'1100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10156678,"num_objects":1240,"num_object_clones":0,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11186,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1016}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11313","reported_seq":15232,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:36.063199+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:36.063199+0000","last_peered":"2024-09-15T21:13:36.063199+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:36.063199+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9730694,"num_objects":1188,"num_object_clones":0,"num_object_copies":4752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11312,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1917,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":817}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11260","reported_seq":11323,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.053177+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:35.053177+0000","last_peered":"2024-09-15T21:13:35.053177+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:35.053177+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9598220,"num_objects":1172,"num_object_clones":0,"num_object_copies":4688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11258,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1107}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11425","reported_seq":11514,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:35.990075+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:35.990075+0000","last_peered":"2024-09-15T21:13:35.990075+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:35.990075+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10286348,"num_objects":1256,"num_object_clones":0,"num_object_copies":5024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11423,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11381","reported_seq":11454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.969615+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:37.969615+0000","last_peered":"2024-09-15T21:13:37.969615+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:37.969615+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":155181602,"num_objects":18948,"num_object_clones":0,"num_object_copies":75792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30525,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18948,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166130,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1938,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311095296,"data_stored":311095296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160875,"ondisk_log_size":160875,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":731044,"kb_used_data":77396,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93640796,"statfs":{"total":96636764160,"available":95888175104,"internally_reserved":0,"allocated":79253504,"data_stored":79080402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3,1,0,1,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721060,"kb_used_data":76244,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93650780,"statfs":{"total":96636764160,"available":95898398720,"internally_reserved":0,"allocated":78073856,"data_stored":77900754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,7,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051699,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575264,"kb_used_data":78544,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93796576,"statfs":{"total":96636764160,"available":96047693824,"internally_reserved":0,"allocated":80429056,"data_stored":80255954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,4,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741996,"kb_used_data":74504,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93629844,"statfs":{"total":96636764160,"available":95876960256,"internally_reserved":0,"allocated":76292096,"data_stored":76118994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,6,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75923456,"data_stored":75923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":79417344,"data_stored":79417344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77463552,"data_stored":77463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":78290944,"data_stored":78290944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:42.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:43.340 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:43.341 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:43.358 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-15T21:13:42.172279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149144098,"num_objects":18211,"num_object_clones":0,"num_object_copies":72844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166867,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753488,"kb_used_data":294968,"kb_used_omap":14,"kb_used_meta":2458417,"kb_avail":374733872,"statfs":{"total":386547056640,"available":383727484928,"internally_reserved":0,"allocated":302047232,"data_stored":301354824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2517419570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,5,9,14,12,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004526"},"pg_stats":[{"pgid":"1.f","version":"35'11259","reported_seq":11311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.050068+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:40.050068+0000","last_peered":"2024-09-15T21:13:40.050068+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:40.050068+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8953856,"num_objects":1093,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11420","reported_seq":16314,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:38.883067+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:38.883067+0000","last_peered":"2024-09-15T21:13:38.883067+0000","last_clean":"2024-09-15T21:13:38.883067+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:38.883067+0000","last_undegraded":"2024-09-15T21:13:38.883067+0000","last_fullsized":"2024-09-15T21:13:38.883067+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9256960,"num_objects":1130,"num_object_clones":0,"num_object_copies":4520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11259","reported_seq":11345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.960725+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:40.960725+0000","last_peered":"2024-09-15T21:13:40.960725+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:40.960725+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607838,"num_objects":1174,"num_object_clones":0,"num_object_copies":4696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11258,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1084}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11327","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.978559+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:37.978559+0000","last_peered":"2024-09-15T21:13:37.978559+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:37.978559+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11539","reported_seq":11602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.025092+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:40.025092+0000","last_peered":"2024-09-15T21:13:40.025092+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:40.025092+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172236,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11290","reported_seq":11341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955991+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:39.955991+0000","last_peered":"2024-09-15T21:13:39.955991+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:39.955991+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8896512,"num_objects":1086,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11604","reported_seq":11681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.944736+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:37.944736+0000","last_peered":"2024-09-15T21:13:37.944736+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:37.944736+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1198}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11596","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.837751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:37.837751+0000","last_peered":"2024-09-15T21:13:37.837751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:37.837751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540876,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11322","reported_seq":11388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955567+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:39.955567+0000","last_peered":"2024-09-15T21:13:39.955567+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:39.955567+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9251352,"num_objects":1130,"num_object_clones":0,"num_object_copies":4520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11433","reported_seq":11520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.008047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:41.008047+0000","last_peered":"2024-09-15T21:13:41.008047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:41.008047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11239","reported_seq":15909,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:40.016365+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:40.016365+0000","last_peered":"2024-09-15T21:13:40.016365+0000","last_clean":"2024-09-15T21:13:40.016365+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:40.016365+0000","last_undegraded":"2024-09-15T21:13:40.016365+0000","last_fullsized":"2024-09-15T21:13:40.016365+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11245","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.038068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:40.038068+0000","last_peered":"2024-09-15T21:13:40.038068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:40.038068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681542,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11382","reported_seq":15427,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.068181+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:41.068181+0000","last_peered":"2024-09-15T21:13:41.068181+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:41.068181+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9165446,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11321","reported_seq":11383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.036706+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:40.036706+0000","last_peered":"2024-09-15T21:13:40.036706+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:40.036706+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098508,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.931395+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:40.931395+0000","last_peered":"2024-09-15T21:13:40.931395+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:40.931395+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712908,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1095}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11381","reported_seq":11454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.969615+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:37.969615+0000","last_peered":"2024-09-15T21:13:37.969615+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:37.969615+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149144098,"num_objects":18211,"num_object_clones":0,"num_object_copies":72844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166867,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299003904,"data_stored":299003904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714704,"kb_used_data":73532,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93657136,"statfs":{"total":96636764160,"available":95904907264,"internally_reserved":0,"allocated":75296768,"data_stored":75123666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721328,"kb_used_data":72352,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93650512,"statfs":{"total":96636764160,"available":95898124288,"internally_reserved":0,"allocated":74088448,"data_stored":73915346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575460,"kb_used_data":74580,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93796380,"statfs":{"total":96636764160,"available":96047493120,"internally_reserved":0,"allocated":76369920,"data_stored":76196818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,1,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741996,"kb_used_data":74504,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93629844,"statfs":{"total":96636764160,"available":95876960256,"internally_reserved":0,"allocated":76292096,"data_stored":76118994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,6,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75923456,"data_stored":75923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75337728,"data_stored":75337728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73441280,"data_stored":73441280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74301440,"data_stored":74301440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:43.359 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:43.943 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:43.945 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:43.968 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-15T21:13:42.172279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149144098,"num_objects":18211,"num_object_clones":0,"num_object_copies":72844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166867,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753488,"kb_used_data":294968,"kb_used_omap":14,"kb_used_meta":2458417,"kb_avail":374733872,"statfs":{"total":386547056640,"available":383727484928,"internally_reserved":0,"allocated":302047232,"data_stored":301354824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2517419570},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,5,9,14,12,16,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18259968,"num_objects":-2229,"num_object_clones":0,"num_object_copies":-8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-186,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2229,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004526"},"pg_stats":[{"pgid":"1.f","version":"35'11259","reported_seq":11311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.050068+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:40.050068+0000","last_peered":"2024-09-15T21:13:40.050068+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:40.050068+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8953856,"num_objects":1093,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11420","reported_seq":16314,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:38.883067+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:38.883067+0000","last_peered":"2024-09-15T21:13:38.883067+0000","last_clean":"2024-09-15T21:13:38.883067+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:38.883067+0000","last_undegraded":"2024-09-15T21:13:38.883067+0000","last_fullsized":"2024-09-15T21:13:38.883067+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9256960,"num_objects":1130,"num_object_clones":0,"num_object_copies":4520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2086,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11259","reported_seq":11345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.960725+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:40.960725+0000","last_peered":"2024-09-15T21:13:40.960725+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:40.960725+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607838,"num_objects":1174,"num_object_clones":0,"num_object_copies":4696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11258,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1084}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11327","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.978559+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:37.978559+0000","last_peered":"2024-09-15T21:13:37.978559+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:37.978559+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11325,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1205}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11539","reported_seq":11602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.025092+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:40.025092+0000","last_peered":"2024-09-15T21:13:40.025092+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:40.025092+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172236,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11290","reported_seq":11341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955991+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:39.955991+0000","last_peered":"2024-09-15T21:13:39.955991+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:39.955991+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8896512,"num_objects":1086,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11604","reported_seq":11681,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.944736+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:37.944736+0000","last_peered":"2024-09-15T21:13:37.944736+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:37.944736+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11602,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1198}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11596","reported_seq":11672,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.837751+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:37.837751+0000","last_peered":"2024-09-15T21:13:37.837751+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:37.837751+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9540876,"num_objects":1165,"num_object_clones":0,"num_object_copies":4660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11594,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1309}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11322","reported_seq":11388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955567+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:39.955567+0000","last_peered":"2024-09-15T21:13:39.955567+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:39.955567+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9251352,"num_objects":1130,"num_object_clones":0,"num_object_copies":4520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11433","reported_seq":11520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.008047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:41.008047+0000","last_peered":"2024-09-15T21:13:41.008047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:41.008047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11239","reported_seq":15909,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:40.016365+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:40.016365+0000","last_peered":"2024-09-15T21:13:40.016365+0000","last_clean":"2024-09-15T21:13:40.016365+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:40.016365+0000","last_undegraded":"2024-09-15T21:13:40.016365+0000","last_fullsized":"2024-09-15T21:13:40.016365+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11245","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.038068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:40.038068+0000","last_peered":"2024-09-15T21:13:40.038068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:40.038068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681542,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11382","reported_seq":15427,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.068181+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:41.068181+0000","last_peered":"2024-09-15T21:13:41.068181+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:41.068181+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9165446,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11321","reported_seq":11383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.036706+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:40.036706+0000","last_peered":"2024-09-15T21:13:40.036706+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:40.036706+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098508,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.931395+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:40.931395+0000","last_peered":"2024-09-15T21:13:40.931395+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:40.931395+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712908,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1095}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11381","reported_seq":11454,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:37.969615+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:37.969615+0000","last_peered":"2024-09-15T21:13:37.969615+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:37.969615+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9565452,"num_objects":1168,"num_object_clones":0,"num_object_copies":4672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11379,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1147}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149144098,"num_objects":18211,"num_object_clones":0,"num_object_copies":72844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18211,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166867,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299003904,"data_stored":299003904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160812,"ondisk_log_size":160812,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714704,"kb_used_data":73532,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93657136,"statfs":{"total":96636764160,"available":95904907264,"internally_reserved":0,"allocated":75296768,"data_stored":75123666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721328,"kb_used_data":72352,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93650512,"statfs":{"total":96636764160,"available":95898124288,"internally_reserved":0,"allocated":74088448,"data_stored":73915346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051700,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575460,"kb_used_data":74580,"kb_used_omap":4,"kb_used_meta":500859,"kb_avail":93796380,"statfs":{"total":96636764160,"available":96047493120,"internally_reserved":0,"allocated":76369920,"data_stored":76196818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":512880335},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,1,5,5,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247938,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":741996,"kb_used_data":74504,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93629844,"statfs":{"total":96636764160,"available":95876960256,"internally_reserved":0,"allocated":76292096,"data_stored":76118994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,2,6,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":75923456,"data_stored":75923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75337728,"data_stored":75337728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73441280,"data_stored":73441280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74301440,"data_stored":74301440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11259", 'reported_seq': 11311, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.050068+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:40.050068+0000', 'last_peered': '2024-09-15T21:13:40.050068+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:40.050068+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8953856, 'num_objects': 1093, 'num_object_clones': 0, 'num_object_copies': 4372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1093, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11259, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1183}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11259", 'reported_seq': 11345, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.960725+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:40.960725+0000', 'last_peered': '2024-09-15T21:13:40.960725+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:40.960725+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9607838, 'num_objects': 1174, 'num_object_clones': 0, 'num_object_copies': 4696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1174, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11258, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1084}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11327", 'reported_seq': 11401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.978559+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:37.978559+0000', 'last_peered': '2024-09-15T21:13:37.978559+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:37.978559+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9245964, 'num_objects': 1129, 'num_object_clones': 0, 'num_object_copies': 4516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11325, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1205}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11539", 'reported_seq': 11602, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.025092+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:40.025092+0000', 'last_peered': '2024-09-15T21:13:40.025092+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:40.025092+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9172236, 'num_objects': 1120, 'num_object_clones': 0, 'num_object_copies': 4480, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1120, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11537, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1191}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:43.970 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11290", 'reported_seq': 11341, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:39.955991+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:39.955991+0000', 'last_peered': '2024-09-15T21:13:39.955991+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:39.955991+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8896512, 'num_objects': 1086, 'num_object_clones': 0, 'num_object_copies': 4344, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1086, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11290, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1230}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11604", 'reported_seq': 11681, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.944736+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:37.944736+0000', 'last_peered': '2024-09-15T21:13:37.944736+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:37.944736+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11602, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1198}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11596", 'reported_seq': 11672, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.837751+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:37.837751+0000', 'last_peered': '2024-09-15T21:13:37.837751+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:37.837751+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9540876, 'num_objects': 1165, 'num_object_clones': 0, 'num_object_copies': 4660, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1165, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11594, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1309}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11322", 'reported_seq': 11388, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:39.955567+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:39.955567+0000', 'last_peered': '2024-09-15T21:13:39.955567+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:39.955567+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9251352, 'num_objects': 1130, 'num_object_clones': 0, 'num_object_copies': 4520, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1130, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11319, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1202}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11433", 'reported_seq': 11520, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:41.008047+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:41.008047+0000', 'last_peered': '2024-09-15T21:13:41.008047+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:41.008047+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9245964, 'num_objects': 1129, 'num_object_clones': 0, 'num_object_copies': 4516, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1129, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11431, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1206}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11245", 'reported_seq': 11291, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.038068+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:40.038068+0000', 'last_peered': '2024-09-15T21:13:40.038068+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:40.038068+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1200", 'ondisk_log_start': "27'1200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9681542, 'num_objects': 1182, 'num_object_clones': 0, 'num_object_copies': 4728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1182, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11244, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.971 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11382", 'reported_seq': 15427, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:41.068181+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:41.068181+0000', 'last_peered': '2024-09-15T21:13:41.068181+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:41.068181+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9165446, 'num_objects': 1119, 'num_object_clones': 0, 'num_object_copies': 4476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 308, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1119, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11381, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1980, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11321", 'reported_seq': 11383, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.036706+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:40.036706+0000', 'last_peered': '2024-09-15T21:13:40.036706+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:40.036706+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9098508, 'num_objects': 1111, 'num_object_clones': 0, 'num_object_copies': 4444, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1111, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11319, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1168}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11495", 'reported_seq': 11584, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:40.931395+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:40.931395+0000', 'last_peered': '2024-09-15T21:13:40.931395+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:40.931395+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9712908, 'num_objects': 1186, 'num_object_clones': 0, 'num_object_copies': 4744, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1186, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11493, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1095}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:43.972 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11381", 'reported_seq': 11454, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:37.969615+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:37.969615+0000', 'last_peered': '2024-09-15T21:13:37.969615+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:37.969615+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9565452, 'num_objects': 1168, 'num_object_clones': 0, 'num_object_copies': 4672, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1168, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11379, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1147}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:43.972 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:44.761 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:44.761 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30462/71620 objects degraded (42.533%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":690,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":17905,"data_bytes":146637346,"bytes_used":2819772416,"bytes_avail":383727284224,"bytes_total":386547056640,"degraded_objects":30462,"degraded_total":71620,"degraded_ratio":0.42532812063669367,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":189},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:44.787 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:44.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:45.370 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:45.371 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:45.393 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-15T21:13:44.172891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146637346,"num_objects":17905,"num_object_clones":0,"num_object_copies":71620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17905,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167173,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2753684,"kb_used_data":286860,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374733676,"statfs":{"total":386547056640,"available":383727284224,"internally_reserved":0,"allocated":293744640,"data_stored":293052232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2525939250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,10,5,11,19,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18653184,"num_objects":-2277,"num_object_clones":0,"num_object_copies":-9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004282"},"pg_stats":[{"pgid":"1.f","version":"35'11259","reported_seq":11311,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.050068+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:40.050068+0000","last_peered":"2024-09-15T21:13:40.050068+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:40.050068+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8953856,"num_objects":1093,"num_object_clones":0,"num_object_copies":4372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11259,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1183}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11472","reported_seq":16366,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:43.821520+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:43.821520+0000","last_peered":"2024-09-15T21:13:43.821520+0000","last_clean":"2024-09-15T21:13:43.821520+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:43.821520+0000","last_undegraded":"2024-09-15T21:13:43.821520+0000","last_fullsized":"2024-09-15T21:13:43.821520+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8830976,"num_objects":1078,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11259","reported_seq":11345,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.960725+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:40.960725+0000","last_peered":"2024-09-15T21:13:40.960725+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:40.960725+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9607838,"num_objects":1174,"num_object_clones":0,"num_object_copies":4696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11258,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1084}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11388","reported_seq":11464,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:42.937331+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:42.937331+0000","last_peered":"2024-09-15T21:13:42.937331+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:42.937331+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8746252,"num_objects":1068,"num_object_clones":0,"num_object_copies":4272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11386,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1266}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11539","reported_seq":11602,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.025092+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:40.025092+0000","last_peered":"2024-09-15T21:13:40.025092+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:40.025092+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9172236,"num_objects":1120,"num_object_clones":0,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11537,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11290","reported_seq":11341,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955991+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:39.955991+0000","last_peered":"2024-09-15T21:13:39.955991+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:39.955991+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8896512,"num_objects":1086,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11290,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1230}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11668","reported_seq":11745,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:42.602321+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:42.602321+0000","last_peered":"2024-09-15T21:13:42.602321+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:42.602321+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9041164,"num_objects":1104,"num_object_clones":0,"num_object_copies":4416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11666,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11672","reported_seq":11748,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:42.946683+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:42.946683+0000","last_peered":"2024-09-15T21:13:42.946683+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:42.946683+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8918284,"num_objects":1089,"num_object_clones":0,"num_object_copies":4356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1385}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11322","reported_seq":11388,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:39.955567+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:39.955567+0000","last_peered":"2024-09-15T21:13:39.955567+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:39.955567+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9251352,"num_objects":1130,"num_object_clones":0,"num_object_copies":4520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11433","reported_seq":11520,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.008047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:41.008047+0000","last_peered":"2024-09-15T21:13:41.008047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:41.008047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9245964,"num_objects":1129,"num_object_clones":0,"num_object_copies":4516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1206}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11239","reported_seq":15909,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:40.016365+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:40.016365+0000","last_peered":"2024-09-15T21:13:40.016365+0000","last_clean":"2024-09-15T21:13:40.016365+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:40.016365+0000","last_undegraded":"2024-09-15T21:13:40.016365+0000","last_fullsized":"2024-09-15T21:13:40.016365+0000","mapping_epoch":32,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9183232,"num_objects":1121,"num_object_clones":0,"num_object_copies":4484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11245","reported_seq":11291,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.038068+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:40.038068+0000","last_peered":"2024-09-15T21:13:40.038068+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:40.038068+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1200","ondisk_log_start":"27'1200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9681542,"num_objects":1182,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11244,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11382","reported_seq":15427,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:41.068181+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:41.068181+0000","last_peered":"2024-09-15T21:13:41.068181+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:41.068181+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9165446,"num_objects":1119,"num_object_clones":0,"num_object_copies":4476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11381,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1980,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11321","reported_seq":11383,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.036706+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:40.036706+0000","last_peered":"2024-09-15T21:13:40.036706+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:40.036706+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9098508,"num_objects":1111,"num_object_clones":0,"num_object_copies":4444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11319,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1168}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11495","reported_seq":11584,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:40.931395+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:40.931395+0000","last_peered":"2024-09-15T21:13:40.931395+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:40.931395+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9712908,"num_objects":1186,"num_object_clones":0,"num_object_copies":4744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11493,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1095}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11434","reported_seq":11509,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:43.048016+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:43.048016+0000","last_peered":"2024-09-15T21:13:43.048016+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:43.048016+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9131276,"num_objects":1115,"num_object_clones":0,"num_object_copies":4460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11432,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1200}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":146637346,"num_objects":17905,"num_object_clones":0,"num_object_copies":71620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17905,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167173,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2001,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":290791424,"data_stored":290791424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160918,"ondisk_log_size":160918,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714704,"kb_used_data":73532,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93657136,"statfs":{"total":96636764160,"available":95904907264,"internally_reserved":0,"allocated":75296768,"data_stored":75123666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,7,2,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721328,"kb_used_data":72352,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93650512,"statfs":{"total":96636764160,"available":95898124288,"internally_reserved":0,"allocated":74088448,"data_stored":73915346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,2,6,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575556,"kb_used_data":70516,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93796284,"statfs":{"total":96636764160,"available":96047394816,"internally_reserved":0,"allocated":72208384,"data_stored":72035282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":742096,"kb_used_data":70460,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93629744,"statfs":{"total":96636764160,"available":95876857856,"internally_reserved":0,"allocated":72151040,"data_stored":71977938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":687730382},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,6,4,6],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":71802880,"data_stored":71802880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71245824,"data_stored":71245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73441280,"data_stored":73441280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":74301440,"data_stored":74301440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:48.395 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:49.048 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:49.048 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:49.078 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-15T21:13:48.174291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139166242,"num_objects":16993,"num_object_clones":0,"num_object_copies":67972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168085,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160630,"ondisk_log_size":160630,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737772,"kb_used_data":275112,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374749588,"statfs":{"total":386547056640,"available":383743578112,"internally_reserved":0,"allocated":281714688,"data_stored":281022280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,18,6,19,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17948672,"num_objects":-2191,"num_object_clones":0,"num_object_copies":-8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004502"},"pg_stats":[{"pgid":"1.f","version":"35'11326","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.949238+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:44.949238+0000","last_peered":"2024-09-15T21:13:44.949238+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:44.949238+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404992,"num_objects":1026,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11472","reported_seq":16366,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:43.821520+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:43.821520+0000","last_peered":"2024-09-15T21:13:43.821520+0000","last_clean":"2024-09-15T21:13:43.821520+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:43.821520+0000","last_undegraded":"2024-09-15T21:13:43.821520+0000","last_fullsized":"2024-09-15T21:13:43.821520+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8830976,"num_objects":1078,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11315","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.959497+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:45.959497+0000","last_peered":"2024-09-15T21:13:45.959497+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:45.959497+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9149086,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11314,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11452","reported_seq":11528,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.970002+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:47.970002+0000","last_peered":"2024-09-15T21:13:47.970002+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:47.970002+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8221964,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11607","reported_seq":11670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.120729+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:45.120729+0000","last_peered":"2024-09-15T21:13:45.120729+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:45.120729+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8615180,"num_objects":1052,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11355","reported_seq":11406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.944673+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:44.944673+0000","last_peered":"2024-09-15T21:13:44.944673+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:44.944673+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364032,"num_objects":1021,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11355,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11725","reported_seq":11802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.999201+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:47.999201+0000","last_peered":"2024-09-15T21:13:47.999201+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:47.999201+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8574220,"num_objects":1047,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11725","reported_seq":11801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.862435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:47.862435+0000","last_peered":"2024-09-15T21:13:47.862435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:47.862435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11377","reported_seq":11444,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.955950+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:44.955950+0000","last_peered":"2024-09-15T21:13:44.955950+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:44.955950+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8800792,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11504","reported_seq":11592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.128173+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:46.128173+0000","last_peered":"2024-09-15T21:13:46.128173+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:46.128173+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8664332,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11307","reported_seq":15975,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:45.039173+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:45.039173+0000","last_peered":"2024-09-15T21:13:45.039173+0000","last_clean":"2024-09-15T21:13:45.039173+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:45.039173+0000","last_undegraded":"2024-09-15T21:13:45.039173+0000","last_fullsized":"2024-09-15T21:13:45.039173+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11305","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.623610+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:44.623610+0000","last_peered":"2024-09-15T21:13:44.623610+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:44.623610+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9190022,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11304,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11439","reported_seq":15624,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.111428+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:46.111428+0000","last_peered":"2024-09-15T21:13:46.111428+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:46.111428+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8698502,"num_objects":1062,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11382","reported_seq":11445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.865537+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:44.865537+0000","last_peered":"2024-09-15T21:13:44.865537+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:44.865537+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598796,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11549","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.960856+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:45.960856+0000","last_peered":"2024-09-15T21:13:45.960856+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:45.960856+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270540,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11490","reported_seq":11565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.946002+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:47.946002+0000","last_peered":"2024-09-15T21:13:47.946002+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:47.946002+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672524,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139166242,"num_objects":16993,"num_object_clones":0,"num_object_copies":67972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168085,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278777856,"data_stored":278777856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160630,"ondisk_log_size":160630,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714876,"kb_used_data":69548,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93656964,"statfs":{"total":96636764160,"available":95904731136,"internally_reserved":0,"allocated":71217152,"data_stored":71044050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,1,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721556,"kb_used_data":68420,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93650284,"statfs":{"total":96636764160,"available":95897890816,"internally_reserved":0,"allocated":70062080,"data_stored":69888978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575556,"kb_used_data":70516,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93796284,"statfs":{"total":96636764160,"available":96047394816,"internally_reserved":0,"allocated":72208384,"data_stored":72035282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725784,"kb_used_data":66628,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93646056,"statfs":{"total":96636764160,"available":95893561344,"internally_reserved":0,"allocated":68227072,"data_stored":68053970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67842048,"data_stored":67842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71245824,"data_stored":71245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69414912,"data_stored":69414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70275072,"data_stored":70275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:49.079 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:49.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:49.664 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:49.692 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-15T21:13:48.174291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":139166242,"num_objects":16993,"num_object_clones":0,"num_object_copies":67972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168085,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160630,"ondisk_log_size":160630,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737772,"kb_used_data":275112,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374749588,"statfs":{"total":386547056640,"available":383743578112,"internally_reserved":0,"allocated":281714688,"data_stored":281022280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,7,4,18,6,19,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17948672,"num_objects":-2191,"num_object_clones":0,"num_object_copies":-8764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004502"},"pg_stats":[{"pgid":"1.f","version":"35'11326","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.949238+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:44.949238+0000","last_peered":"2024-09-15T21:13:44.949238+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:44.949238+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404992,"num_objects":1026,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11472","reported_seq":16366,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:43.821520+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:43.821520+0000","last_peered":"2024-09-15T21:13:43.821520+0000","last_clean":"2024-09-15T21:13:43.821520+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:43.821520+0000","last_undegraded":"2024-09-15T21:13:43.821520+0000","last_fullsized":"2024-09-15T21:13:43.821520+0000","mapping_epoch":34,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8830976,"num_objects":1078,"num_object_clones":0,"num_object_copies":4312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2138,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11315","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.959497+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:45.959497+0000","last_peered":"2024-09-15T21:13:45.959497+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:45.959497+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9149086,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11314,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11452","reported_seq":11528,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.970002+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:47.970002+0000","last_peered":"2024-09-15T21:13:47.970002+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:47.970002+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8221964,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11607","reported_seq":11670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.120729+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:45.120729+0000","last_peered":"2024-09-15T21:13:45.120729+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:45.120729+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8615180,"num_objects":1052,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11355","reported_seq":11406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.944673+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:44.944673+0000","last_peered":"2024-09-15T21:13:44.944673+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:44.944673+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364032,"num_objects":1021,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11355,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11725","reported_seq":11802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.999201+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:47.999201+0000","last_peered":"2024-09-15T21:13:47.999201+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:47.999201+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8574220,"num_objects":1047,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11725","reported_seq":11801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.862435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:47.862435+0000","last_peered":"2024-09-15T21:13:47.862435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:47.862435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11377","reported_seq":11444,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.955950+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:44.955950+0000","last_peered":"2024-09-15T21:13:44.955950+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:44.955950+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8800792,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11504","reported_seq":11592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.128173+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:46.128173+0000","last_peered":"2024-09-15T21:13:46.128173+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:46.128173+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8664332,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11307","reported_seq":15975,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:45.039173+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:45.039173+0000","last_peered":"2024-09-15T21:13:45.039173+0000","last_clean":"2024-09-15T21:13:45.039173+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:45.039173+0000","last_undegraded":"2024-09-15T21:13:45.039173+0000","last_fullsized":"2024-09-15T21:13:45.039173+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11305","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.623610+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:44.623610+0000","last_peered":"2024-09-15T21:13:44.623610+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:44.623610+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9190022,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11304,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11439","reported_seq":15624,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.111428+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:46.111428+0000","last_peered":"2024-09-15T21:13:46.111428+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:46.111428+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8698502,"num_objects":1062,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11382","reported_seq":11445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.865537+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:44.865537+0000","last_peered":"2024-09-15T21:13:44.865537+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:44.865537+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598796,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11549","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.960856+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:45.960856+0000","last_peered":"2024-09-15T21:13:45.960856+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:45.960856+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270540,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11490","reported_seq":11565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.946002+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:47.946002+0000","last_peered":"2024-09-15T21:13:47.946002+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:47.946002+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672524,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":139166242,"num_objects":16993,"num_object_clones":0,"num_object_copies":67972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16993,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168085,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":278777856,"data_stored":278777856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160630,"ondisk_log_size":160630,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714876,"kb_used_data":69548,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93656964,"statfs":{"total":96636764160,"available":95904731136,"internally_reserved":0,"allocated":71217152,"data_stored":71044050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,1,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721556,"kb_used_data":68420,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93650284,"statfs":{"total":96636764160,"available":95897890816,"internally_reserved":0,"allocated":70062080,"data_stored":69888978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051701,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575556,"kb_used_data":70516,"kb_used_omap":4,"kb_used_meta":505019,"kb_avail":93796284,"statfs":{"total":96636764160,"available":96047394816,"internally_reserved":0,"allocated":72208384,"data_stored":72035282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":517140175},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,2,2,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725784,"kb_used_data":66628,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93646056,"statfs":{"total":96636764160,"available":95893561344,"internally_reserved":0,"allocated":68227072,"data_stored":68053970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67842048,"data_stored":67842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":71245824,"data_stored":71245824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69414912,"data_stored":69414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70275072,"data_stored":70275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:49.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:49.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11326", 'reported_seq': 11378, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:44.949238+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:44.949238+0000', 'last_peered': '2024-09-15T21:13:44.949238+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:44.949238+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8404992, 'num_objects': 1026, 'num_object_clones': 0, 'num_object_copies': 4104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1026, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11326, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1250}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:49.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11315", 'reported_seq': 11401, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:45.959497+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:45.959497+0000', 'last_peered': '2024-09-15T21:13:45.959497+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:45.959497+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9149086, 'num_objects': 1118, 'num_object_clones': 0, 'num_object_copies': 4472, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1118, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11314, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1140}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11452", 'reported_seq': 11528, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:47.970002+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:47.970002+0000', 'last_peered': '2024-09-15T21:13:47.970002+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:47.970002+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8221964, 'num_objects': 1004, 'num_object_clones': 0, 'num_object_copies': 4016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11450, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1330}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11607", 'reported_seq': 11670, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:45.120729+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:45.120729+0000', 'last_peered': '2024-09-15T21:13:45.120729+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:45.120729+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8615180, 'num_objects': 1052, 'num_object_clones': 0, 'num_object_copies': 4208, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1052, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11605, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1259}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11355", 'reported_seq': 11406, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:44.944673+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:44.944673+0000', 'last_peered': '2024-09-15T21:13:44.944673+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:44.944673+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10055, 'log_dups_size': 0, 'ondisk_log_size': 10055, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8364032, 'num_objects': 1021, 'num_object_clones': 0, 'num_object_copies': 4084, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1021, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11355, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1295}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11725", 'reported_seq': 11802, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:47.999201+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:47.999201+0000', 'last_peered': '2024-09-15T21:13:47.999201+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:47.999201+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8574220, 'num_objects': 1047, 'num_object_clones': 0, 'num_object_copies': 4188, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1047, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11723, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1319}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11725", 'reported_seq': 11801, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:47.862435+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:47.862435+0000', 'last_peered': '2024-09-15T21:13:47.862435+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:47.862435+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8484108, 'num_objects': 1036, 'num_object_clones': 0, 'num_object_copies': 4144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1036, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11723, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1438}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:49.694 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11377", 'reported_seq': 11444, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:44.955950+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:44.955950+0000', 'last_peered': '2024-09-15T21:13:44.955950+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:44.955950+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8800792, 'num_objects': 1075, 'num_object_clones': 0, 'num_object_copies': 4300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11374, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1257}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11504", 'reported_seq': 11592, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:46.128173+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:46.128173+0000', 'last_peered': '2024-09-15T21:13:46.128173+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:46.128173+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8664332, 'num_objects': 1058, 'num_object_clones': 0, 'num_object_copies': 4232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1058, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11502, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1277}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11305", 'reported_seq': 11351, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:44.623610+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:44.623610+0000', 'last_peered': '2024-09-15T21:13:44.623610+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:44.623610+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9190022, 'num_objects': 1122, 'num_object_clones': 0, 'num_object_copies': 4488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1122, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11304, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1134}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11439", 'reported_seq': 15624, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:46.111428+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:46.111428+0000', 'last_peered': '2024-09-15T21:13:46.111428+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:46.111428+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8698502, 'num_objects': 1062, 'num_object_clones': 0, 'num_object_copies': 4248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 238, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1062, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11438, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2050, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 824}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11382", 'reported_seq': 11445, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:44.865537+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:44.865537+0000', 'last_peered': '2024-09-15T21:13:44.865537+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:44.865537+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8598796, 'num_objects': 1050, 'num_object_clones': 0, 'num_object_copies': 4200, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1050, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11380, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1229}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:49.695 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11549", 'reported_seq': 11638, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:45.960856+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:45.960856+0000', 'last_peered': '2024-09-15T21:13:45.960856+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:45.960856+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9270540, 'num_objects': 1132, 'num_object_clones': 0, 'num_object_copies': 4528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1132, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11547, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1149}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:49.696 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:49.696 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11490", 'reported_seq': 11565, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:47.946002+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:47.946002+0000', 'last_peered': '2024-09-15T21:13:47.946002+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:47.946002+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8672524, 'num_objects': 1059, 'num_object_clones': 0, 'num_object_copies': 4236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1059, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11488, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1256}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:49.696 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:50.468 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:50.469 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30392/67760 objects degraded (44.852%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":696,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16940,"data_bytes":138732066,"bytes_used":2803036160,"bytes_avail":383744020480,"bytes_total":386547056640,"degraded_objects":30392,"degraded_total":67760,"degraded_ratio":0.44852420306965762,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":167},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:50.498 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:50.499 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:51.135 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:51.135 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:51.156 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-09-15T21:13:50.175171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138732066,"num_objects":16940,"num_object_clones":0,"num_object_copies":67760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168138,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160583,"ondisk_log_size":160583,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2737340,"kb_used_data":270508,"kb_used_omap":14,"kb_used_meta":2466737,"kb_avail":374750020,"statfs":{"total":386547056640,"available":383744020480,"internally_reserved":0,"allocated":277000192,"data_stored":276307784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2525939250},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,3,20,7,19,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16449536,"num_objects":-2008,"num_object_clones":0,"num_object_copies":-8032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2008,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":133,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004676"},"pg_stats":[{"pgid":"1.f","version":"35'11326","reported_seq":11378,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.949238+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:44.949238+0000","last_peered":"2024-09-15T21:13:44.949238+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:44.949238+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8404992,"num_objects":1026,"num_object_clones":0,"num_object_copies":4104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11525","reported_seq":16419,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:49.056971+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:49.056971+0000","last_peered":"2024-09-15T21:13:49.056971+0000","last_clean":"2024-09-15T21:13:49.056971+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:49.056971+0000","last_undegraded":"2024-09-15T21:13:49.056971+0000","last_fullsized":"2024-09-15T21:13:49.056971+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8396800,"num_objects":1025,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11315","reported_seq":11401,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.959497+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:45.959497+0000","last_peered":"2024-09-15T21:13:45.959497+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:45.959497+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9149086,"num_objects":1118,"num_object_clones":0,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1118,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11314,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11452","reported_seq":11528,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.970002+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:47.970002+0000","last_peered":"2024-09-15T21:13:47.970002+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:47.970002+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8221964,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11450,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1330}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11607","reported_seq":11670,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.120729+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:45.120729+0000","last_peered":"2024-09-15T21:13:45.120729+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:45.120729+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8615180,"num_objects":1052,"num_object_clones":0,"num_object_copies":4208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1259}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11355","reported_seq":11406,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.944673+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:44.944673+0000","last_peered":"2024-09-15T21:13:44.944673+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:44.944673+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8364032,"num_objects":1021,"num_object_clones":0,"num_object_copies":4084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11355,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1295}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11725","reported_seq":11802,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.999201+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:47.999201+0000","last_peered":"2024-09-15T21:13:47.999201+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:47.999201+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8574220,"num_objects":1047,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1319}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11725","reported_seq":11801,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.862435+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:47.862435+0000","last_peered":"2024-09-15T21:13:47.862435+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:47.862435+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8484108,"num_objects":1036,"num_object_clones":0,"num_object_copies":4144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11723,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1438}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11377","reported_seq":11444,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.955950+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:44.955950+0000","last_peered":"2024-09-15T21:13:44.955950+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:44.955950+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8800792,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1257}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11504","reported_seq":11592,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.128173+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:46.128173+0000","last_peered":"2024-09-15T21:13:46.128173+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:46.128173+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8664332,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11502,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1277}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11307","reported_seq":15975,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:45.039173+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:45.039173+0000","last_peered":"2024-09-15T21:13:45.039173+0000","last_clean":"2024-09-15T21:13:45.039173+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:45.039173+0000","last_undegraded":"2024-09-15T21:13:45.039173+0000","last_fullsized":"2024-09-15T21:13:45.039173+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8626176,"num_objects":1053,"num_object_clones":0,"num_object_copies":4212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11305","reported_seq":11351,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.623610+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:44.623610+0000","last_peered":"2024-09-15T21:13:44.623610+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:44.623610+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9190022,"num_objects":1122,"num_object_clones":0,"num_object_copies":4488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11304,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1134}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11439","reported_seq":15624,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:46.111428+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:46.111428+0000","last_peered":"2024-09-15T21:13:46.111428+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:46.111428+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8698502,"num_objects":1062,"num_object_clones":0,"num_object_copies":4248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":238,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11438,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2050,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":824}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11382","reported_seq":11445,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:44.865537+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:44.865537+0000","last_peered":"2024-09-15T21:13:44.865537+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:44.865537+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8598796,"num_objects":1050,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1229}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11549","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:45.960856+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:45.960856+0000","last_peered":"2024-09-15T21:13:45.960856+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:45.960856+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9270540,"num_objects":1132,"num_object_clones":0,"num_object_copies":4528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11547,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11490","reported_seq":11565,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:47.946002+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:47.946002+0000","last_peered":"2024-09-15T21:13:47.946002+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:47.946002+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8672524,"num_objects":1059,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11488,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1256}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":138732066,"num_objects":16940,"num_object_clones":0,"num_object_copies":67760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30392,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168138,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2071,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274620416,"data_stored":274620416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160583,"ondisk_log_size":160583,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":714876,"kb_used_data":69548,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93656964,"statfs":{"total":96636764160,"available":95904731136,"internally_reserved":0,"allocated":71217152,"data_stored":71044050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,5,1,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":721556,"kb_used_data":68420,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93650284,"statfs":{"total":96636764160,"available":95897890816,"internally_reserved":0,"allocated":70062080,"data_stored":69888978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575124,"kb_used_data":65912,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93796716,"statfs":{"total":96636764160,"available":96047837184,"internally_reserved":0,"allocated":67493888,"data_stored":67320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247940,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725784,"kb_used_data":66628,"kb_used_omap":4,"kb_used_meta":659131,"kb_avail":93646056,"statfs":{"total":96636764160,"available":95893561344,"internally_reserved":0,"allocated":68227072,"data_stored":68053970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":674950862},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,2,8,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":67842048,"data_stored":67842048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67088384,"data_stored":67088384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69414912,"data_stored":69414912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70275072,"data_stored":70275072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:54.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:54.771 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:54.771 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:54.803 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-15T21:13:54.176826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131056162,"num_objects":16003,"num_object_clones":0,"num_object_copies":64012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169075,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721492,"kb_used_data":258788,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374765868,"statfs":{"total":386547056640,"available":383760248832,"internally_reserved":0,"allocated":264998912,"data_stored":264306504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,17,13,9,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"1.f","version":"35'11380","reported_seq":11432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.098589+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:50.098589+0000","last_peered":"2024-09-15T21:13:50.098589+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:50.098589+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7962624,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11525","reported_seq":16419,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:49.056971+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:49.056971+0000","last_peered":"2024-09-15T21:13:49.056971+0000","last_clean":"2024-09-15T21:13:49.056971+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:49.056971+0000","last_undegraded":"2024-09-15T21:13:49.056971+0000","last_fullsized":"2024-09-15T21:13:49.056971+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8396800,"num_objects":1025,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11375","reported_seq":11461,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.087613+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:51.087613+0000","last_peered":"2024-09-15T21:13:51.087613+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:51.087613+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8657566,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11374,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11514","reported_seq":11589,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.910114+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:52.910114+0000","last_peered":"2024-09-15T21:13:52.910114+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:52.910114+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11672","reported_seq":11735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.077384+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:50.077384+0000","last_peered":"2024-09-15T21:13:50.077384+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:50.077384+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8082700,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11411","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:49.986353+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:49.986353+0000","last_peered":"2024-09-15T21:13:49.986353+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:49.986353+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7905280,"num_objects":965,"num_object_clones":0,"num_object_copies":3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11792","reported_seq":11869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.084993+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:53.084993+0000","last_peered":"2024-09-15T21:13:53.084993+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:53.084993+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025356,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11790","reported_seq":11866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.993278+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:52.993278+0000","last_peered":"2024-09-15T21:13:52.993278+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:52.993278+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951628,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11434","reported_seq":11501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.127209+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:50.127209+0000","last_peered":"2024-09-15T21:13:50.127209+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:50.127209+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8333848,"num_objects":1018,"num_object_clones":0,"num_object_copies":4072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11574","reported_seq":11662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.692446+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:50.692446+0000","last_peered":"2024-09-15T21:13:50.692446+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:50.692446+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090892,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11377","reported_seq":16047,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:49.985143+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:49.985143+0000","last_peered":"2024-09-15T21:13:49.985143+0000","last_clean":"2024-09-15T21:13:49.985143+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:49.985143+0000","last_undegraded":"2024-09-15T21:13:49.985143+0000","last_fullsized":"2024-09-15T21:13:49.985143+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11378","reported_seq":11424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.085398+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:50.085398+0000","last_peered":"2024-09-15T21:13:50.085398+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:50.085398+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8592006,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11507","reported_seq":15803,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.163794+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:51.163794+0000","last_peered":"2024-09-15T21:13:51.163794+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:51.163794+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11438","reported_seq":11501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.067037+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:50.067037+0000","last_peered":"2024-09-15T21:13:50.067037+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:50.067037+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8140044,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1285}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11606","reported_seq":11696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.225065+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:51.225065+0000","last_peered":"2024-09-15T21:13:51.225065+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:51.225065+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8803596,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11547","reported_seq":11622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.118278+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:53.118278+0000","last_peered":"2024-09-15T21:13:53.118278+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:53.118278+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8205580,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":131056162,"num_objects":16003,"num_object_clones":0,"num_object_copies":64012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169075,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262578176,"data_stored":262578176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715068,"kb_used_data":65576,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93656772,"statfs":{"total":96636764160,"available":95904534528,"internally_reserved":0,"allocated":67149824,"data_stored":66976722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4,7,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705144,"kb_used_data":64476,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93666696,"statfs":{"total":96636764160,"available":95914696704,"internally_reserved":0,"allocated":66023424,"data_stored":65850322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575124,"kb_used_data":65912,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93796716,"statfs":{"total":96636764160,"available":96047837184,"internally_reserved":0,"allocated":67493888,"data_stored":67320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726156,"kb_used_data":62824,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93645684,"statfs":{"total":96636764160,"available":95893180416,"internally_reserved":0,"allocated":64331776,"data_stored":64158674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63827968,"data_stored":63827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67088384,"data_stored":67088384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65441792,"data_stored":65441792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66220032,"data_stored":66220032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:54.804 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:55.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:55.408 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:55.443 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-15T21:13:54.176826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131056162,"num_objects":16003,"num_object_clones":0,"num_object_copies":64012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169075,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2721492,"kb_used_data":258788,"kb_used_omap":14,"kb_used_meta":2462577,"kb_avail":374765868,"statfs":{"total":386547056640,"available":383760248832,"internally_reserved":0,"allocated":264998912,"data_stored":264306504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2521679410},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,17,13,9,10,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18087936,"num_objects":-2208,"num_object_clones":0,"num_object_copies":-8832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004547"},"pg_stats":[{"pgid":"1.f","version":"35'11380","reported_seq":11432,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.098589+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:50.098589+0000","last_peered":"2024-09-15T21:13:50.098589+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:50.098589+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7962624,"num_objects":972,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11380,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1304}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11525","reported_seq":16419,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:49.056971+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:49.056971+0000","last_peered":"2024-09-15T21:13:49.056971+0000","last_clean":"2024-09-15T21:13:49.056971+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:49.056971+0000","last_undegraded":"2024-09-15T21:13:49.056971+0000","last_fullsized":"2024-09-15T21:13:49.056971+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8396800,"num_objects":1025,"num_object_clones":0,"num_object_copies":4100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2191,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11375","reported_seq":11461,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.087613+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:51.087613+0000","last_peered":"2024-09-15T21:13:51.087613+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:51.087613+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8657566,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11374,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11514","reported_seq":11589,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.910114+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:52.910114+0000","last_peered":"2024-09-15T21:13:52.910114+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:52.910114+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11672","reported_seq":11735,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.077384+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:50.077384+0000","last_peered":"2024-09-15T21:13:50.077384+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:50.077384+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8082700,"num_objects":987,"num_object_clones":0,"num_object_copies":3948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1324}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11411","reported_seq":11462,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:49.986353+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:49.986353+0000","last_peered":"2024-09-15T21:13:49.986353+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:49.986353+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7905280,"num_objects":965,"num_object_clones":0,"num_object_copies":3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11411,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1351}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11792","reported_seq":11869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.084993+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:53.084993+0000","last_peered":"2024-09-15T21:13:53.084993+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:53.084993+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025356,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11790","reported_seq":11866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.993278+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:52.993278+0000","last_peered":"2024-09-15T21:13:52.993278+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:52.993278+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951628,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11434","reported_seq":11501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.127209+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:50.127209+0000","last_peered":"2024-09-15T21:13:50.127209+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:50.127209+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8333848,"num_objects":1018,"num_object_clones":0,"num_object_copies":4072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11574","reported_seq":11662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.692446+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:50.692446+0000","last_peered":"2024-09-15T21:13:50.692446+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:50.692446+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090892,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11377","reported_seq":16047,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:49.985143+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:49.985143+0000","last_peered":"2024-09-15T21:13:49.985143+0000","last_clean":"2024-09-15T21:13:49.985143+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:49.985143+0000","last_undegraded":"2024-09-15T21:13:49.985143+0000","last_fullsized":"2024-09-15T21:13:49.985143+0000","mapping_epoch":32,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8052736,"num_objects":983,"num_object_clones":0,"num_object_copies":3932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5488,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11378","reported_seq":11424,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.085398+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:50.085398+0000","last_peered":"2024-09-15T21:13:50.085398+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:50.085398+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8592006,"num_objects":1049,"num_object_clones":0,"num_object_copies":4196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1207}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11507","reported_seq":15803,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.163794+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:51.163794+0000","last_peered":"2024-09-15T21:13:51.163794+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:51.163794+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11438","reported_seq":11501,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.067037+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:50.067037+0000","last_peered":"2024-09-15T21:13:50.067037+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:50.067037+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8140044,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11436,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1285}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11606","reported_seq":11696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.225065+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:51.225065+0000","last_peered":"2024-09-15T21:13:51.225065+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:51.225065+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8803596,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11547","reported_seq":11622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.118278+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:53.118278+0000","last_peered":"2024-09-15T21:13:53.118278+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:53.118278+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8205580,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":131056162,"num_objects":16003,"num_object_clones":0,"num_object_copies":64012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16003,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169075,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262578176,"data_stored":262578176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160820,"ondisk_log_size":160820,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715068,"kb_used_data":65576,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93656772,"statfs":{"total":96636764160,"available":95904534528,"internally_reserved":0,"allocated":67149824,"data_stored":66976722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,0,4,7,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705144,"kb_used_data":64476,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93666696,"statfs":{"total":96636764160,"available":95914696704,"internally_reserved":0,"allocated":66023424,"data_stored":65850322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051702,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575124,"kb_used_data":65912,"kb_used_omap":4,"kb_used_meta":509179,"kb_avail":93796716,"statfs":{"total":96636764160,"available":96047837184,"internally_reserved":0,"allocated":67493888,"data_stored":67320786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":521400015},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,3,4,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726156,"kb_used_data":62824,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93645684,"statfs":{"total":96636764160,"available":95893180416,"internally_reserved":0,"allocated":64331776,"data_stored":64158674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63827968,"data_stored":63827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67088384,"data_stored":67088384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65441792,"data_stored":65441792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":66220032,"data_stored":66220032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11380", 'reported_seq': 11432, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.098589+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:50.098589+0000', 'last_peered': '2024-09-15T21:13:50.098589+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:50.098589+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7962624, 'num_objects': 972, 'num_object_clones': 0, 'num_object_copies': 3888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 972, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11380, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1304}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11375", 'reported_seq': 11461, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:51.087613+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:51.087613+0000', 'last_peered': '2024-09-15T21:13:51.087613+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:51.087613+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8657566, 'num_objects': 1058, 'num_object_clones': 0, 'num_object_copies': 4232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1058, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11374, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1200}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:13:55.444 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11514", 'reported_seq': 11589, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:52.910114+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:52.910114+0000', 'last_peered': '2024-09-15T21:13:52.910114+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:52.910114+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7714060, 'num_objects': 942, 'num_object_clones': 0, 'num_object_copies': 3768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11512, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1392}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:55.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:13:55.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11672", 'reported_seq': 11735, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.077384+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:50.077384+0000', 'last_peered': '2024-09-15T21:13:50.077384+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:50.077384+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8082700, 'num_objects': 987, 'num_object_clones': 0, 'num_object_copies': 3948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 987, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11670, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1324}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.445 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:13:55.445 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11411", 'reported_seq': 11462, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:49.986353+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:49.986353+0000', 'last_peered': '2024-09-15T21:13:49.986353+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:49.986353+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7905280, 'num_objects': 965, 'num_object_clones': 0, 'num_object_copies': 3860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11411, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1351}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11792", 'reported_seq': 11869, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:53.084993+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:53.084993+0000', 'last_peered': '2024-09-15T21:13:53.084993+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:53.084993+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8025356, 'num_objects': 980, 'num_object_clones': 0, 'num_object_copies': 3920, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 980, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11790, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1386}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11790", 'reported_seq': 11866, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:52.993278+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:52.993278+0000', 'last_peered': '2024-09-15T21:13:52.993278+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:52.993278+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7951628, 'num_objects': 971, 'num_object_clones': 0, 'num_object_copies': 3884, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 971, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11788, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1503}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11434", 'reported_seq': 11501, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.127209+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:50.127209+0000', 'last_peered': '2024-09-15T21:13:50.127209+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:50.127209+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8333848, 'num_objects': 1018, 'num_object_clones': 0, 'num_object_copies': 4072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1018, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11431, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1314}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:13:55.446 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11574", 'reported_seq': 11662, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.692446+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:50.692446+0000', 'last_peered': '2024-09-15T21:13:50.692446+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:50.692446+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8090892, 'num_objects': 988, 'num_object_clones': 0, 'num_object_copies': 3952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 988, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11572, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1347}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11378", 'reported_seq': 11424, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.085398+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:50.085398+0000', 'last_peered': '2024-09-15T21:13:50.085398+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:50.085398+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1300", 'ondisk_log_start': "27'1300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10078, 'log_dups_size': 0, 'ondisk_log_size': 10078, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8592006, 'num_objects': 1049, 'num_object_clones': 0, 'num_object_copies': 4196, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1049, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11377, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1207}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11507", 'reported_seq': 15803, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:51.163794+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:51.163794+0000', 'last_peered': '2024-09-15T21:13:51.163794+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:51.163794+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8141446, 'num_objects': 994, 'num_object_clones': 0, 'num_object_copies': 3976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 183, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11506, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2105, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 811}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11438", 'reported_seq': 11501, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:50.067037+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:50.067037+0000', 'last_peered': '2024-09-15T21:13:50.067037+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:50.067037+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10038, 'log_dups_size': 0, 'ondisk_log_size': 10038, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8140044, 'num_objects': 994, 'num_object_clones': 0, 'num_object_copies': 3976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 994, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11436, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1285}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11606", 'reported_seq': 11696, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:51.225065+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:51.225065+0000', 'last_peered': '2024-09-15T21:13:51.225065+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:51.225065+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8803596, 'num_objects': 1075, 'num_object_clones': 0, 'num_object_copies': 4300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1075, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11604, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1206}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:13:55.447 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11547", 'reported_seq': 11622, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:53.118278+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:53.118278+0000', 'last_peered': '2024-09-15T21:13:53.118278+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:53.118278+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8205580, 'num_objects': 1002, 'num_object_clones': 0, 'num_object_copies': 4008, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1002, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11545, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1313}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:13:55.448 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:13:56.186 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:56.186 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30337/64012 objects degraded (47.393%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":701,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16003,"data_bytes":131056162,"bytes_used":2786807808,"bytes_avail":383760248832,"bytes_total":386547056640,"degraded_objects":30337,"degraded_total":64012,"degraded_ratio":0.47392676373180026,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":183},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:13:56.207 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:13:56.208 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:13:56.809 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:13:56.810 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:13:56.838 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-15T21:13:56.177614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":127132194,"num_objects":15524,"num_object_clones":0,"num_object_copies":62096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15524,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169554,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705208,"kb_used_data":250840,"kb_used_omap":14,"kb_used_meta":2454257,"kb_avail":374782152,"statfs":{"total":386547056640,"available":383776923648,"internally_reserved":0,"allocated":256860160,"data_stored":256167752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2513159730},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,2,15,9,11,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19505152,"num_objects":-2381,"num_object_clones":0,"num_object_copies":-9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-125,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2381,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004722"},"pg_stats":[{"pgid":"1.f","version":"35'11435","reported_seq":11487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.219018+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:55.219018+0000","last_peered":"2024-09-15T21:13:55.219018+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:55.219018+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7512064,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11582","reported_seq":16476,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:54.218049+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:54.218049+0000","last_peered":"2024-09-15T21:13:54.218049+0000","last_clean":"2024-09-15T21:13:54.218049+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:54.218049+0000","last_undegraded":"2024-09-15T21:13:54.218049+0000","last_fullsized":"2024-09-15T21:13:54.218049+0000","mapping_epoch":34,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":968,"num_object_clones":0,"num_object_copies":3872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2248,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11375","reported_seq":11461,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.087613+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:51.087613+0000","last_peered":"2024-09-15T21:13:51.087613+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:51.087613+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1300","ondisk_log_start":"27'1300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8657566,"num_objects":1058,"num_object_clones":0,"num_object_copies":4232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11374,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11514","reported_seq":11589,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.910114+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:52.910114+0000","last_peered":"2024-09-15T21:13:52.910114+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:52.910114+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11512,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1392}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11727","reported_seq":11790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.161823+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:55.161823+0000","last_peered":"2024-09-15T21:13:55.161823+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:55.161823+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7632140,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11475","reported_seq":11526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.097944+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:55.097944+0000","last_peered":"2024-09-15T21:13:55.097944+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:55.097944+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11475,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11792","reported_seq":11869,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.084993+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:53.084993+0000","last_peered":"2024-09-15T21:13:53.084993+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:53.084993+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8025356,"num_objects":980,"num_object_clones":0,"num_object_copies":3920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11790,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1386}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11790","reported_seq":11866,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:52.993278+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:52.993278+0000","last_peered":"2024-09-15T21:13:52.993278+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:52.993278+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7951628,"num_objects":971,"num_object_clones":0,"num_object_copies":3884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11788,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1503}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11487","reported_seq":11554,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.094991+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:55.094991+0000","last_peered":"2024-09-15T21:13:55.094991+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:55.094991+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899672,"num_objects":965,"num_object_clones":0,"num_object_copies":3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11574","reported_seq":11662,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:50.692446+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:50.692446+0000","last_peered":"2024-09-15T21:13:50.692446+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:50.692446+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8090892,"num_objects":988,"num_object_clones":0,"num_object_copies":3952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11572,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1347}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11440","reported_seq":16110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:55.152726+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:55.152726+0000","last_peered":"2024-09-15T21:13:55.152726+0000","last_clean":"2024-09-15T21:13:55.152726+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:55.152726+0000","last_undegraded":"2024-09-15T21:13:55.152726+0000","last_fullsized":"2024-09-15T21:13:55.152726+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":920,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11443","reported_seq":11489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.095221+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:55.095221+0000","last_peered":"2024-09-15T21:13:55.095221+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:55.095221+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8059526,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11507","reported_seq":15803,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.163794+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:51.163794+0000","last_peered":"2024-09-15T21:13:51.163794+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:51.163794+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8141446,"num_objects":994,"num_object_clones":0,"num_object_copies":3976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11506,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2105,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11505","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.190515+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:55.190515+0000","last_peered":"2024-09-15T21:13:55.190515+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:55.190515+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7591180,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11503,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11606","reported_seq":11696,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:51.225065+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:51.225065+0000","last_peered":"2024-09-15T21:13:51.225065+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:51.225065+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8803596,"num_objects":1075,"num_object_clones":0,"num_object_copies":4300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1206}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11547","reported_seq":11622,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:53.118278+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:53.118278+0000","last_peered":"2024-09-15T21:13:53.118278+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:53.118278+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8205580,"num_objects":1002,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1313}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":127132194,"num_objects":15524,"num_object_clones":0,"num_object_copies":62096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15524,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169554,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2126,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":254369792,"data_stored":254369792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160799,"ondisk_log_size":160799,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715284,"kb_used_data":61636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93656556,"statfs":{"total":96636764160,"available":95904313344,"internally_reserved":0,"allocated":63115264,"data_stored":62942162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705144,"kb_used_data":64476,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93666696,"statfs":{"total":96636764160,"available":95914696704,"internally_reserved":0,"allocated":66023424,"data_stored":65850322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,2,1,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051703,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":558624,"kb_used_data":61904,"kb_used_omap":4,"kb_used_meta":496699,"kb_avail":93813216,"statfs":{"total":96636764160,"available":96064733184,"internally_reserved":0,"allocated":63389696,"data_stored":63216594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":508620495},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,3,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247941,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726156,"kb_used_data":62824,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93645684,"statfs":{"total":96636764160,"available":95893180416,"internally_reserved":0,"allocated":64331776,"data_stored":64158674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,3,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":63827968,"data_stored":63827968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":62984192,"data_stored":62984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65441792,"data_stored":65441792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62115840,"data_stored":62115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:13:59.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:00.461 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:00.462 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:00.485 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-15T21:14:00.179324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122339874,"num_objects":14939,"num_object_clones":0,"num_object_copies":59756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170139,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2724016,"kb_used_data":239152,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374763344,"statfs":{"total":386547056640,"available":383757664256,"internally_reserved":0,"allocated":244891648,"data_stored":244199240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2544420402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,5,9,16,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005032"},"pg_stats":[{"pgid":"1.f","version":"35'11435","reported_seq":11487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.219018+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:55.219018+0000","last_peered":"2024-09-15T21:13:55.219018+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:55.219018+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7512064,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11644","reported_seq":16538,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:59.144246+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:59.144246+0000","last_peered":"2024-09-15T21:13:59.144246+0000","last_clean":"2024-09-15T21:13:59.144246+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:59.144246+0000","last_undegraded":"2024-09-15T21:13:59.144246+0000","last_fullsized":"2024-09-15T21:13:59.144246+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7421952,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11435","reported_seq":11521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.064428+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:56.064428+0000","last_peered":"2024-09-15T21:13:56.064428+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:56.064428+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8166046,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11434,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11584","reported_seq":11660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.200022+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:58.200022+0000","last_peered":"2024-09-15T21:13:58.200022+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:58.200022+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7140620,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11727","reported_seq":11790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.161823+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:55.161823+0000","last_peered":"2024-09-15T21:13:55.161823+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:55.161823+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7632140,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11475","reported_seq":11526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.097944+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:55.097944+0000","last_peered":"2024-09-15T21:13:55.097944+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:55.097944+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11475,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11865","reported_seq":11942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.067030+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:58.067030+0000","last_peered":"2024-09-15T21:13:58.067030+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:58.067030+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7427340,"num_objects":907,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11847","reported_seq":11923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.140909+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:58.140909+0000","last_peered":"2024-09-15T21:13:58.140909+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:58.140909+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484684,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11487","reported_seq":11554,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.094991+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:55.094991+0000","last_peered":"2024-09-15T21:13:55.094991+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:55.094991+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899672,"num_objects":965,"num_object_clones":0,"num_object_copies":3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11636","reported_seq":11724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.231396+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:56.231396+0000","last_peered":"2024-09-15T21:13:56.231396+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:56.231396+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7582988,"num_objects":926,"num_object_clones":0,"num_object_copies":3704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11634,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11440","reported_seq":16110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:55.152726+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:55.152726+0000","last_peered":"2024-09-15T21:13:55.152726+0000","last_clean":"2024-09-15T21:13:55.152726+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:55.152726+0000","last_undegraded":"2024-09-15T21:13:55.152726+0000","last_fullsized":"2024-09-15T21:13:55.152726+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":920,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11443","reported_seq":11489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.095221+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:55.095221+0000","last_peered":"2024-09-15T21:13:55.095221+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:55.095221+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8059526,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11577","reported_seq":15981,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.230809+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:56.230809+0000","last_peered":"2024-09-15T21:13:56.230809+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:56.230809+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7568006,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11505","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.190515+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:55.190515+0000","last_peered":"2024-09-15T21:13:55.190515+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:55.190515+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7591180,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11503,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11677","reported_seq":11766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.142966+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:56.142966+0000","last_peered":"2024-09-15T21:13:56.142966+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:56.142966+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8221964,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11675,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11607","reported_seq":11682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.181189+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:58.181189+0000","last_peered":"2024-09-15T21:13:58.181189+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:58.181189+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122339874,"num_objects":14939,"num_object_clones":0,"num_object_copies":59756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170139,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":242442240,"data_stored":242442240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715284,"kb_used_data":61636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93656556,"statfs":{"total":96636764160,"available":95904313344,"internally_reserved":0,"allocated":63115264,"data_stored":62942162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705348,"kb_used_data":60532,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93666492,"statfs":{"total":96636764160,"available":95914487808,"internally_reserved":0,"allocated":61984768,"data_stored":61811666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576908,"kb_used_data":57980,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93794932,"statfs":{"total":96636764160,"available":96046010368,"internally_reserved":0,"allocated":59371520,"data_stored":59198418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726476,"kb_used_data":59004,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93645364,"statfs":{"total":96636764160,"available":95892852736,"internally_reserved":0,"allocated":60420096,"data_stored":60246994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,2,4,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":59858944,"data_stored":59858944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59039744,"data_stored":59039744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61427712,"data_stored":61427712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62115840,"data_stored":62115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:00.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:01.078 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:01.078 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:01.110 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-15T21:14:00.179324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122339874,"num_objects":14939,"num_object_clones":0,"num_object_copies":59756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170139,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2724016,"kb_used_data":239152,"kb_used_omap":14,"kb_used_meta":2484785,"kb_avail":374763344,"statfs":{"total":386547056640,"available":383757664256,"internally_reserved":0,"allocated":244891648,"data_stored":244199240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2544420402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,0,5,9,16,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-16826368,"num_objects":-2054,"num_object_clones":0,"num_object_copies":-8216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2054,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005032"},"pg_stats":[{"pgid":"1.f","version":"35'11435","reported_seq":11487,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.219018+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:13:55.219018+0000","last_peered":"2024-09-15T21:13:55.219018+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:13:55.219018+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7512064,"num_objects":917,"num_object_clones":0,"num_object_copies":3668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11435,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1359}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11644","reported_seq":16538,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:59.144246+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:59.144246+0000","last_peered":"2024-09-15T21:13:59.144246+0000","last_clean":"2024-09-15T21:13:59.144246+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:59.144246+0000","last_undegraded":"2024-09-15T21:13:59.144246+0000","last_fullsized":"2024-09-15T21:13:59.144246+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7421952,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11435","reported_seq":11521,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.064428+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:13:56.064428+0000","last_peered":"2024-09-15T21:13:56.064428+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:13:56.064428+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8166046,"num_objects":998,"num_object_clones":0,"num_object_copies":3992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11434,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11584","reported_seq":11660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.200022+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:58.200022+0000","last_peered":"2024-09-15T21:13:58.200022+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:58.200022+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7140620,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11727","reported_seq":11790,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.161823+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:13:55.161823+0000","last_peered":"2024-09-15T21:13:55.161823+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:13:55.161823+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7632140,"num_objects":932,"num_object_clones":0,"num_object_copies":3728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11725,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1379}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11475","reported_seq":11526,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.097944+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:13:55.097944+0000","last_peered":"2024-09-15T21:13:55.097944+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:13:55.097944+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7380992,"num_objects":901,"num_object_clones":0,"num_object_copies":3604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11475,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1415}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11865","reported_seq":11942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.067030+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:58.067030+0000","last_peered":"2024-09-15T21:13:58.067030+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:58.067030+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7427340,"num_objects":907,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11847","reported_seq":11923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.140909+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:58.140909+0000","last_peered":"2024-09-15T21:13:58.140909+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:58.140909+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484684,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11487","reported_seq":11554,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.094991+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:13:55.094991+0000","last_peered":"2024-09-15T21:13:55.094991+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:13:55.094991+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899672,"num_objects":965,"num_object_clones":0,"num_object_copies":3860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11484,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1367}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11636","reported_seq":11724,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.231396+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:13:56.231396+0000","last_peered":"2024-09-15T21:13:56.231396+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:13:56.231396+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7582988,"num_objects":926,"num_object_clones":0,"num_object_copies":3704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11634,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1409}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11440","reported_seq":16110,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:55.152726+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:13:55.152726+0000","last_peered":"2024-09-15T21:13:55.152726+0000","last_clean":"2024-09-15T21:13:55.152726+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:13:55.152726+0000","last_undegraded":"2024-09-15T21:13:55.152726+0000","last_fullsized":"2024-09-15T21:13:55.152726+0000","mapping_epoch":32,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":920,"num_object_clones":0,"num_object_copies":3680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5551,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11443","reported_seq":11489,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.095221+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:13:55.095221+0000","last_peered":"2024-09-15T21:13:55.095221+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:13:55.095221+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8059526,"num_objects":984,"num_object_clones":0,"num_object_copies":3936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1272}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11577","reported_seq":15981,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.230809+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:13:56.230809+0000","last_peered":"2024-09-15T21:13:56.230809+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:13:56.230809+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7568006,"num_objects":924,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11576,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2159,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":795}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11505","reported_seq":11567,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:55.190515+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:55.190515+0000","last_peered":"2024-09-15T21:13:55.190515+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:55.190515+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7591180,"num_objects":927,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11503,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1352}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11677","reported_seq":11766,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:56.142966+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:13:56.142966+0000","last_peered":"2024-09-15T21:13:56.142966+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:13:56.142966+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8221964,"num_objects":1004,"num_object_clones":0,"num_object_copies":4016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11675,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1277}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11607","reported_seq":11682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.181189+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:58.181189+0000","last_peered":"2024-09-15T21:13:58.181189+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:58.181189+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122339874,"num_objects":14939,"num_object_clones":0,"num_object_copies":59756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14939,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170139,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2180,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":242442240,"data_stored":242442240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160784,"ondisk_log_size":160784,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":715284,"kb_used_data":61636,"kb_used_omap":4,"kb_used_meta":653627,"kb_avail":93656556,"statfs":{"total":96636764160,"available":95904313344,"internally_reserved":0,"allocated":63115264,"data_stored":62942162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":669314768},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705348,"kb_used_data":60532,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93666492,"statfs":{"total":96636764160,"available":95914487808,"internally_reserved":0,"allocated":61984768,"data_stored":61811666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":660273605},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,3,3,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576908,"kb_used_data":57980,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93794932,"statfs":{"total":96636764160,"available":96046010368,"internally_reserved":0,"allocated":59371520,"data_stored":59198418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726476,"kb_used_data":59004,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93645364,"statfs":{"total":96636764160,"available":95892852736,"internally_reserved":0,"allocated":60420096,"data_stored":60246994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,2,4,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":59858944,"data_stored":59858944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59039744,"data_stored":59039744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61427712,"data_stored":61427712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":62115840,"data_stored":62115840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:01.111 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:01.111 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11435", 'reported_seq': 11487, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.219018+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:13:55.219018+0000', 'last_peered': '2024-09-15T21:13:55.219018+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:13:55.219018+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7512064, 'num_objects': 917, 'num_object_clones': 0, 'num_object_copies': 3668, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 917, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11435, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1359}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.111 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11435", 'reported_seq': 11521, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:56.064428+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:13:56.064428+0000', 'last_peered': '2024-09-15T21:13:56.064428+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:13:56.064428+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10035, 'log_dups_size': 0, 'ondisk_log_size': 10035, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8166046, 'num_objects': 998, 'num_object_clones': 0, 'num_object_copies': 3992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 998, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11434, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1260}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11584", 'reported_seq': 11660, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:58.200022+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:13:58.200022+0000', 'last_peered': '2024-09-15T21:13:58.200022+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:13:58.200022+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10084, 'log_dups_size': 0, 'ondisk_log_size': 10084, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7140620, 'num_objects': 872, 'num_object_clones': 0, 'num_object_copies': 3488, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 872, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11582, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1462}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11727", 'reported_seq': 11790, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.161823+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:13:55.161823+0000', 'last_peered': '2024-09-15T21:13:55.161823+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:13:55.161823+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7632140, 'num_objects': 932, 'num_object_clones': 0, 'num_object_copies': 3728, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 932, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11725, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1379}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11475", 'reported_seq': 11526, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.097944+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:13:55.097944+0000', 'last_peered': '2024-09-15T21:13:55.097944+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:13:55.097944+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7380992, 'num_objects': 901, 'num_object_clones': 0, 'num_object_copies': 3604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 901, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11475, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1415}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11865", 'reported_seq': 11942, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:58.067030+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:13:58.067030+0000', 'last_peered': '2024-09-15T21:13:58.067030+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:13:58.067030+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7427340, 'num_objects': 907, 'num_object_clones': 0, 'num_object_copies': 3628, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 907, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11863, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1459}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:01.112 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11847", 'reported_seq': 11923, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:58.140909+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:13:58.140909+0000', 'last_peered': '2024-09-15T21:13:58.140909+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:13:58.140909+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7484684, 'num_objects': 914, 'num_object_clones': 0, 'num_object_copies': 3656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 914, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11845, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1560}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11487", 'reported_seq': 11554, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.094991+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:13:55.094991+0000', 'last_peered': '2024-09-15T21:13:55.094991+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:13:55.094991+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7899672, 'num_objects': 965, 'num_object_clones': 0, 'num_object_copies': 3860, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 965, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11484, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1367}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11636", 'reported_seq': 11724, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:56.231396+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:13:56.231396+0000', 'last_peered': '2024-09-15T21:13:56.231396+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:13:56.231396+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7582988, 'num_objects': 926, 'num_object_clones': 0, 'num_object_copies': 3704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 926, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11634, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1409}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11443", 'reported_seq': 11489, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.095221+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:13:55.095221+0000', 'last_peered': '2024-09-15T21:13:55.095221+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:13:55.095221+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1400", 'ondisk_log_start': "27'1400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8059526, 'num_objects': 984, 'num_object_clones': 0, 'num_object_copies': 3936, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 984, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11442, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1272}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11577", 'reported_seq': 15981, 'reported_epoch': 35, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:56.230809+0000', 'last_change': '2024-09-15T21:10:40.406955+0000', 'last_active': '2024-09-15T21:13:56.230809+0000', 'last_peered': '2024-09-15T21:13:56.230809+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:13:56.230809+0000', 'last_undegraded': '2024-09-15T21:03:59.077418+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7568006, 'num_objects': 924, 'num_object_clones': 0, 'num_object_copies': 3696, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 129, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 924, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11576, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2159, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 795}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:01.113 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:01.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11505", 'reported_seq': 11567, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:55.190515+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:13:55.190515+0000', 'last_peered': '2024-09-15T21:13:55.190515+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:13:55.190515+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7591180, 'num_objects': 927, 'num_object_clones': 0, 'num_object_copies': 3708, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 927, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11503, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1352}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:01.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:01.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11677", 'reported_seq': 11766, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:56.142966+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:13:56.142966+0000', 'last_peered': '2024-09-15T21:13:56.142966+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:13:56.142966+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 8221964, 'num_objects': 1004, 'num_object_clones': 0, 'num_object_copies': 4016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1004, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11675, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:01.114 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:01.114 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11607", 'reported_seq': 11682, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:13:58.181189+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:13:58.181189+0000', 'last_peered': '2024-09-15T21:13:58.181189+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:13:58.181189+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7714060, 'num_objects': 942, 'num_object_clones': 0, 'num_object_copies': 3768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 942, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11605, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1373}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:01.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:01.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:01.843 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30283/59756 objects degraded (50.678%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":707,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":14},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14939,"data_bytes":122339874,"bytes_used":2789392384,"bytes_avail":383757664256,"bytes_total":386547056640,"degraded_objects":30283,"degraded_total":59756,"degraded_ratio":0.5067775620858157,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":109,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":171},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:01.869 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:01.870 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:02.484 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:02.485 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:02.512 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-15T21:14:02.180233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":116892194,"num_objects":14274,"num_object_clones":0,"num_object_copies":57096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14274,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170804,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707896,"kb_used_data":231320,"kb_used_omap":14,"kb_used_meta":2476465,"kb_avail":374779464,"statfs":{"total":386547056640,"available":383774171136,"internally_reserved":0,"allocated":236871680,"data_stored":236179272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2535900722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,0,10,5,13,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21839872,"num_objects":-2666,"num_object_clones":0,"num_object_copies":-10664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005061"},"pg_stats":[{"pgid":"1.f","version":"35'11487","reported_seq":11539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.275373+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:00.275373+0000","last_peered":"2024-09-15T21:14:00.275373+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:00.275373+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7086080,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11487,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11644","reported_seq":16538,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:59.144246+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:59.144246+0000","last_peered":"2024-09-15T21:13:59.144246+0000","last_clean":"2024-09-15T21:13:59.144246+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:59.144246+0000","last_undegraded":"2024-09-15T21:13:59.144246+0000","last_fullsized":"2024-09-15T21:13:59.144246+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7421952,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11493","reported_seq":11577,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.270585+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:01.270585+0000","last_peered":"2024-09-15T21:14:01.270585+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:01.270585+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7690910,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11492,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1318}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11584","reported_seq":11660,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.200022+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:13:58.200022+0000","last_peered":"2024-09-15T21:13:58.200022+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:13:58.200022+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7140620,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11582,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1462}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11778","reported_seq":11841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.316633+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:00.316633+0000","last_peered":"2024-09-15T21:14:00.316633+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:00.316633+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7214348,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11541","reported_seq":11591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.421463+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:00.421463+0000","last_peered":"2024-09-15T21:14:00.421463+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:00.421463+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840320,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11865","reported_seq":11942,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.067030+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:13:58.067030+0000","last_peered":"2024-09-15T21:13:58.067030+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:13:58.067030+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7427340,"num_objects":907,"num_object_clones":0,"num_object_copies":3628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11863,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1459}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11847","reported_seq":11923,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.140909+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:13:58.140909+0000","last_peered":"2024-09-15T21:13:58.140909+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:13:58.140909+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7484684,"num_objects":914,"num_object_clones":0,"num_object_copies":3656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11845,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1560}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11559","reported_seq":11626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.380250+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:00.380250+0000","last_peered":"2024-09-15T21:14:00.380250+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:00.380250+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309848,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11698","reported_seq":11786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.137222+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:01.137222+0000","last_peered":"2024-09-15T21:14:01.137222+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:01.137222+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075084,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11696,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11505","reported_seq":16173,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:00.341286+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:00.341286+0000","last_peered":"2024-09-15T21:14:00.341286+0000","last_clean":"2024-09-15T21:14:00.341286+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:00.341286+0000","last_undegraded":"2024-09-15T21:14:00.341286+0000","last_fullsized":"2024-09-15T21:14:00.341286+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7004160,"num_objects":855,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11517","reported_seq":11563,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.271669+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:00.271669+0000","last_peered":"2024-09-15T21:14:00.271669+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:00.271669+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7453318,"num_objects":910,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11638","reported_seq":16172,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.239092+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:14:01.239092+0000","last_peered":"2024-09-15T21:14:01.239092+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:14:01.239092+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7068294,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":64,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11560","reported_seq":11623,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:59.996005+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:59.996005+0000","last_peered":"2024-09-15T21:13:59.996005+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:59.996005+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7140620,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11726","reported_seq":11816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.160472+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:01.160472+0000","last_peered":"2024-09-15T21:14:01.160472+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:01.160472+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820556,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1326}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11607","reported_seq":11682,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:58.181189+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:13:58.181189+0000","last_peered":"2024-09-15T21:13:58.181189+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:13:58.181189+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7714060,"num_objects":942,"num_object_clones":0,"num_object_copies":3768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11605,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1373}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":116892194,"num_objects":14274,"num_object_clones":0,"num_object_copies":57096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14274,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170804,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234442752,"data_stored":234442752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160849,"ondisk_log_size":160849,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698916,"kb_used_data":57720,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93672924,"statfs":{"total":96636764160,"available":95921074176,"internally_reserved":0,"allocated":59105280,"data_stored":58932178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705596,"kb_used_data":56616,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93666244,"statfs":{"total":96636764160,"available":95914233856,"internally_reserved":0,"allocated":57974784,"data_stored":57801682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576908,"kb_used_data":57980,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93794932,"statfs":{"total":96636764160,"available":96046010368,"internally_reserved":0,"allocated":59371520,"data_stored":59198418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247942,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726476,"kb_used_data":59004,"kb_used_omap":4,"kb_used_meta":667451,"kb_avail":93645364,"statfs":{"total":96636764160,"available":95892852736,"internally_reserved":0,"allocated":60420096,"data_stored":60246994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":683470542},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,0,2,4,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":59858944,"data_stored":59858944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59039744,"data_stored":59039744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57438208,"data_stored":57438208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58105856,"data_stored":58105856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:05.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:06.146 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:06.146 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:06.174 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-15T21:14:04.184086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114958882,"num_objects":14038,"num_object_clones":0,"num_object_copies":56152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171040,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708236,"kb_used_data":227500,"kb_used_omap":14,"kb_used_meta":2480625,"kb_avail":374779124,"statfs":{"total":386547056640,"available":383773822976,"internally_reserved":0,"allocated":232960000,"data_stored":232267592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2540160562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,13,10,12,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18153472,"num_objects":-2216,"num_object_clones":0,"num_object_copies":-8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008086"},"pg_stats":[{"pgid":"1.f","version":"35'11487","reported_seq":11539,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.275373+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:00.275373+0000","last_peered":"2024-09-15T21:14:00.275373+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:00.275373+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7086080,"num_objects":865,"num_object_clones":0,"num_object_copies":3460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11487,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1411}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11644","reported_seq":16538,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:13:59.144246+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:13:59.144246+0000","last_peered":"2024-09-15T21:13:59.144246+0000","last_clean":"2024-09-15T21:13:59.144246+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:13:59.144246+0000","last_undegraded":"2024-09-15T21:13:59.144246+0000","last_fullsized":"2024-09-15T21:13:59.144246+0000","mapping_epoch":34,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7421952,"num_objects":906,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11493","reported_seq":11577,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.270585+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:01.270585+0000","last_peered":"2024-09-15T21:14:01.270585+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:01.270585+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1400","ondisk_log_start":"27'1400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7690910,"num_objects":940,"num_object_clones":0,"num_object_copies":3760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11492,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1318}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11644","reported_seq":11720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.142356+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:03.142356+0000","last_peered":"2024-09-15T21:14:03.142356+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:03.142356+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6649100,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11778","reported_seq":11841,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.316633+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:00.316633+0000","last_peered":"2024-09-15T21:14:00.316633+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:00.316633+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7214348,"num_objects":881,"num_object_clones":0,"num_object_copies":3524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11776,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11541","reported_seq":11591,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.421463+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:00.421463+0000","last_peered":"2024-09-15T21:14:00.421463+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:00.421463+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840320,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11541,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1481}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11928","reported_seq":12005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.922998+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:02.922998+0000","last_peered":"2024-09-15T21:14:02.922998+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:02.922998+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911244,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11904","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.947482+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:02.947482+0000","last_peered":"2024-09-15T21:14:02.947482+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:02.947482+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7017740,"num_objects":857,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11559","reported_seq":11626,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.380250+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:00.380250+0000","last_peered":"2024-09-15T21:14:00.380250+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:00.380250+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7309848,"num_objects":893,"num_object_clones":0,"num_object_copies":3572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1439}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11698","reported_seq":11786,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.137222+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:01.137222+0000","last_peered":"2024-09-15T21:14:01.137222+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:01.137222+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7075084,"num_objects":864,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11696,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1471}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11505","reported_seq":16173,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:00.341286+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:00.341286+0000","last_peered":"2024-09-15T21:14:00.341286+0000","last_clean":"2024-09-15T21:14:00.341286+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:00.341286+0000","last_undegraded":"2024-09-15T21:14:00.341286+0000","last_fullsized":"2024-09-15T21:14:00.341286+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7004160,"num_objects":855,"num_object_clones":0,"num_object_copies":3420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5616,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11517","reported_seq":11563,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:00.271669+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:00.271669+0000","last_peered":"2024-09-15T21:14:00.271669+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:00.271669+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7453318,"num_objects":910,"num_object_clones":0,"num_object_copies":3640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1346}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11638","reported_seq":16172,"reported_epoch":35,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.239092+0000","last_change":"2024-09-15T21:10:40.406955+0000","last_active":"2024-09-15T21:14:01.239092+0000","last_peered":"2024-09-15T21:14:01.239092+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:14:01.239092+0000","last_undegraded":"2024-09-15T21:03:59.077418+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7068294,"num_objects":863,"num_object_clones":0,"num_object_copies":3452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":64,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11637,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2224,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":799}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11560","reported_seq":11623,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:13:59.996005+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:13:59.996005+0000","last_peered":"2024-09-15T21:13:59.996005+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:13:59.996005+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7140620,"num_objects":872,"num_object_clones":0,"num_object_copies":3488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11558,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1407}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11726","reported_seq":11816,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:01.160472+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:01.160472+0000","last_peered":"2024-09-15T21:14:01.160472+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:01.160472+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7820556,"num_objects":955,"num_object_clones":0,"num_object_copies":3820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11724,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1326}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11663","reported_seq":11738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.205968+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:14:03.205968+0000","last_peered":"2024-09-15T21:14:03.205968+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:03.205968+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255308,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":114958882,"num_objects":14038,"num_object_clones":0,"num_object_copies":56152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14038,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171040,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2245,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230551552,"data_stored":230551552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160785,"ondisk_log_size":160785,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698916,"kb_used_data":57720,"kb_used_omap":4,"kb_used_meta":641147,"kb_avail":93672924,"statfs":{"total":96636764160,"available":95921074176,"internally_reserved":0,"allocated":59105280,"data_stored":58932178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":656535248},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,1,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705596,"kb_used_data":56616,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":93666244,"statfs":{"total":96636764160,"available":95914233856,"internally_reserved":0,"allocated":57974784,"data_stored":57801682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051704,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":576908,"kb_used_data":57980,"kb_used_omap":4,"kb_used_meta":518907,"kb_avail":93794932,"statfs":{"total":96636764160,"available":96046010368,"internally_reserved":0,"allocated":59371520,"data_stored":59198418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":531361487},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726816,"kb_used_data":55184,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93645024,"statfs":{"total":96636764160,"available":95892504576,"internally_reserved":0,"allocated":56508416,"data_stored":56335314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":687730382},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,7,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55967744,"data_stored":55967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":59039744,"data_stored":59039744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57438208,"data_stored":57438208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":58105856,"data_stored":58105856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:06.175 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:06.773 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:06.773 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:06.794 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-15T21:14:06.446790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108839458,"num_objects":13291,"num_object_clones":0,"num_object_copies":53164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171787,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708280,"kb_used_data":215080,"kb_used_omap":14,"kb_used_meta":2493105,"kb_avail":374779080,"statfs":{"total":386547056640,"available":383773777920,"internally_reserved":0,"allocated":220241920,"data_stored":219549512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2552940082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,2,10,14,25,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18292736,"num_objects":-2233,"num_object_clones":0,"num_object_copies":-8932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-183,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.269176"},"pg_stats":[{"pgid":"1.f","version":"35'11548","reported_seq":11600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.324898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:05.324898+0000","last_peered":"2024-09-15T21:14:05.324898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:05.324898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11717","reported_seq":16609,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:04.347144+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:04.347144+0000","last_peered":"2024-09-15T21:14:04.347144+0000","last_clean":"2024-09-15T21:14:04.347144+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:04.347144+0000","last_undegraded":"2024-09-15T21:14:04.347144+0000","last_fullsized":"2024-09-15T21:14:04.347144+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823936,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11551","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.016100+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:06.016100+0000","last_peered":"2024-09-15T21:14:06.016100+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:06.016100+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7215774,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11550,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11644","reported_seq":11720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.142356+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:03.142356+0000","last_peered":"2024-09-15T21:14:03.142356+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:03.142356+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6649100,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11846","reported_seq":11907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.401565+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:05.401565+0000","last_peered":"2024-09-15T21:14:05.401565+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:05.401565+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6657292,"num_objects":813,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11596","reported_seq":11647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:04.966072+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:04.966072+0000","last_peered":"2024-09-15T21:14:04.966072+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:04.966072+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6389760,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11928","reported_seq":12005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.922998+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:02.922998+0000","last_peered":"2024-09-15T21:14:02.922998+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:02.922998+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911244,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11904","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.947482+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:02.947482+0000","last_peered":"2024-09-15T21:14:02.947482+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:02.947482+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7017740,"num_objects":857,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11619","reported_seq":11686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.439008+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:05.439008+0000","last_peered":"2024-09-15T21:14:05.439008+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:05.439008+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6818328,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11756","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.180945+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:06.180945+0000","last_peered":"2024-09-15T21:14:06.180945+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:06.180945+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6599948,"num_objects":806,"num_object_clones":0,"num_object_copies":3224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11563","reported_seq":16232,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:05.490932+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:05.490932+0000","last_peered":"2024-09-15T21:14:05.490932+0000","last_clean":"2024-09-15T21:14:05.490932+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:05.490932+0000","last_undegraded":"2024-09-15T21:14:05.490932+0000","last_fullsized":"2024-09-15T21:14:05.490932+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6529024,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11593","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.473528+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:05.473528+0000","last_peered":"2024-09-15T21:14:05.473528+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:05.473528+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6830726,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11707","reported_seq":16370,"reported_epoch":35,"state":"active+undersized+remapped","last_fresh":"2024-09-15T21:14:06.262176+0000","last_change":"2024-09-15T21:14:06.168049+0000","last_active":"2024-09-15T21:14:06.262176+0000","last_peered":"2024-09-15T21:14:06.262176+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:14:06.262176+0000","last_undegraded":"2024-09-15T21:14:06.262176+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6503046,"num_objects":794,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","1(0)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11611","reported_seq":11671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.152534+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:05.152534+0000","last_peered":"2024-09-15T21:14:05.152534+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:05.152534+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722828,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11786","reported_seq":11875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.099305+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:06.099305+0000","last_peered":"2024-09-15T21:14:06.099305+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:06.099305+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7329036,"num_objects":895,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11784,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11663","reported_seq":11738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.205968+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:14:03.205968+0000","last_peered":"2024-09-15T21:14:03.205968+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:03.205968+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255308,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108839458,"num_objects":13291,"num_object_clones":0,"num_object_copies":53164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171787,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218480640,"data_stored":218480640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698448,"kb_used_data":53116,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93673392,"statfs":{"total":96636764160,"available":95921553408,"internally_reserved":0,"allocated":54390784,"data_stored":54217682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705836,"kb_used_data":52700,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93666004,"statfs":{"total":96636764160,"available":95913988096,"internally_reserved":0,"allocated":53964800,"data_stored":53791698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577180,"kb_used_data":54080,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93794660,"statfs":{"total":96636764160,"available":96045731840,"internally_reserved":0,"allocated":55377920,"data_stored":55204818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726816,"kb_used_data":55184,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93645024,"statfs":{"total":96636764160,"available":95892504576,"internally_reserved":0,"allocated":56508416,"data_stored":56335314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":687730382},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,7,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55967744,"data_stored":55967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55062528,"data_stored":55062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53424128,"data_stored":53424128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54026240,"data_stored":54026240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:06.796 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:06.796 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "35'11548", 'reported_seq': 11600, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:05.324898+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:05.324898+0000', 'last_peered': '2024-09-15T21:14:05.324898+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:05.324898+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6586368, 'num_objects': 804, 'num_object_clones': 0, 'num_object_copies': 3216, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 804, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11548, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1472}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:06.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "35'11551", 'reported_seq': 11638, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:06.016100+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:06.016100+0000', 'last_peered': '2024-09-15T21:14:06.016100+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:06.016100+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7215774, 'num_objects': 882, 'num_object_clones': 0, 'num_object_copies': 3528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 882, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11550, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1376}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:06.797 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:06.797 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "35'11644", 'reported_seq': 11720, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:03.142356+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:03.142356+0000', 'last_peered': '2024-09-15T21:14:03.142356+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:03.142356+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10044, 'log_dups_size': 0, 'ondisk_log_size': 10044, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6649100, 'num_objects': 812, 'num_object_clones': 0, 'num_object_copies': 3248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 812, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11642, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1522}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:06.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:06.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "35'11846", 'reported_seq': 11907, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:05.401565+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:05.401565+0000', 'last_peered': '2024-09-15T21:14:05.401565+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:05.401565+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6657292, 'num_objects': 813, 'num_object_clones': 0, 'num_object_copies': 3252, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 813, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11844, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1498}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.798 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:06.798 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "35'11596", 'reported_seq': 11647, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:04.966072+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:04.966072+0000', 'last_peered': '2024-09-15T21:14:04.966072+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:04.966072+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6389760, 'num_objects': 780, 'num_object_clones': 0, 'num_object_copies': 3120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 780, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11596, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1536}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.800 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:06.800 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "35'11928", 'reported_seq': 12005, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:02.922998+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:02.922998+0000', 'last_peered': '2024-09-15T21:14:02.922998+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:02.922998+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6911244, 'num_objects': 844, 'num_object_clones': 0, 'num_object_copies': 3376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 844, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11926, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1522}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "35'11904", 'reported_seq': 11980, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:02.947482+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:02.947482+0000', 'last_peered': '2024-09-15T21:14:02.947482+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:02.947482+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7017740, 'num_objects': 857, 'num_object_clones': 0, 'num_object_copies': 3428, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 857, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11902, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1617}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "35'11619", 'reported_seq': 11686, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:05.439008+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:05.439008+0000', 'last_peered': '2024-09-15T21:14:05.439008+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:05.439008+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6818328, 'num_objects': 833, 'num_object_clones': 0, 'num_object_copies': 3332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 833, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11616, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1499}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "35'11756", 'reported_seq': 11844, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:06.180945+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:06.180945+0000', 'last_peered': '2024-09-15T21:14:06.180945+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:06.180945+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10056, 'log_dups_size': 0, 'ondisk_log_size': 10056, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6599948, 'num_objects': 806, 'num_object_clones': 0, 'num_object_copies': 3224, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 806, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11754, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1529}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "35'11593", 'reported_seq': 11638, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:05.473528+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:05.473528+0000', 'last_peered': '2024-09-15T21:14:05.473528+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:05.473528+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1500", 'ondisk_log_start': "27'1500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10093, 'log_dups_size': 0, 'ondisk_log_size': 10093, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6830726, 'num_objects': 834, 'num_object_clones': 0, 'num_object_copies': 3336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 834, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11592, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1422}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:14:06.801 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "35'11707", 'reported_seq': 16370, 'reported_epoch': 35, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-15T21:14:06.262176+0000', 'last_change': '2024-09-15T21:14:06.168049+0000', 'last_active': '2024-09-15T21:14:06.262176+0000', 'last_peered': '2024-09-15T21:14:06.262176+0000', 'last_clean': '2024-09-15T21:02:42.003716+0000', 'last_became_active': '2024-09-15T21:03:59.668598+0000', 'last_became_peered': '2024-09-15T21:03:59.668598+0000', 'last_unstale': '2024-09-15T21:14:06.262176+0000', 'last_undegraded': '2024-09-15T21:14:06.262176+0000', 'last_fullsized': '2024-09-15T21:03:59.040514+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6503046, 'num_objects': 794, 'num_object_clones': 0, 'num_object_copies': 3176, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 794, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11706, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2288, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '1(0)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': 794}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "35'11611", 'reported_seq': 11671, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:05.152534+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:05.152534+0000', 'last_peered': '2024-09-15T21:14:05.152534+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:05.152534+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6722828, 'num_objects': 821, 'num_object_clones': 0, 'num_object_copies': 3284, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 821, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11609, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1458}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "35'11786", 'reported_seq': 11875, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:06.099305+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:06.099305+0000', 'last_peered': '2024-09-15T21:14:06.099305+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:06.099305+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7329036, 'num_objects': 895, 'num_object_clones': 0, 'num_object_copies': 3580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 895, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11784, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:06.802 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "35'11663", 'reported_seq': 11738, 'reported_epoch': 35, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:03.205968+0000', 'last_change': '2024-09-15T21:03:59.662483+0000', 'last_active': '2024-09-15T21:14:03.205968+0000', 'last_peered': '2024-09-15T21:14:03.205968+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:03.205968+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7255308, 'num_objects': 886, 'num_object_clones': 0, 'num_object_copies': 3544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2317, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 886, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11661, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1429}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:06.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:07.561 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:07.562 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30218/54144 objects degraded (55.810%), 14 pgs degraded, 14 pgs undersized","count":28},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":713,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":13},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":13536,"data_bytes":110846498,"bytes_used":2773032960,"bytes_avail":383774023680,"bytes_total":386547056640,"degraded_objects":30218,"degraded_total":54144,"degraded_ratio":0.55810431442080377,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":205},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:07.588 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:07.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:08.202 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:08.202 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:08.237 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-09-15T21:14:08.185395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108839458,"num_objects":13291,"num_object_clones":0,"num_object_copies":53164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171787,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708280,"kb_used_data":215080,"kb_used_omap":14,"kb_used_meta":2493105,"kb_avail":374779080,"statfs":{"total":386547056640,"available":383773777920,"internally_reserved":0,"allocated":220241920,"data_stored":219549512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2552940082},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,2,10,14,25,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13500416,"num_objects":-1648,"num_object_clones":0,"num_object_copies":-6592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1648,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006071"},"pg_stats":[{"pgid":"1.f","version":"35'11548","reported_seq":11600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.324898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:05.324898+0000","last_peered":"2024-09-15T21:14:05.324898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:05.324898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"35'11717","reported_seq":16609,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:04.347144+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:04.347144+0000","last_peered":"2024-09-15T21:14:04.347144+0000","last_clean":"2024-09-15T21:14:04.347144+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:04.347144+0000","last_undegraded":"2024-09-15T21:14:04.347144+0000","last_fullsized":"2024-09-15T21:14:04.347144+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6823936,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2383,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11551","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.016100+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:06.016100+0000","last_peered":"2024-09-15T21:14:06.016100+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:06.016100+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7215774,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11550,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"35'11644","reported_seq":11720,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.142356+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:03.142356+0000","last_peered":"2024-09-15T21:14:03.142356+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:03.142356+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6649100,"num_objects":812,"num_object_clones":0,"num_object_copies":3248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11642,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1522}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11846","reported_seq":11907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.401565+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:05.401565+0000","last_peered":"2024-09-15T21:14:05.401565+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:05.401565+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6657292,"num_objects":813,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11596","reported_seq":11647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:04.966072+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:04.966072+0000","last_peered":"2024-09-15T21:14:04.966072+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:04.966072+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6389760,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"35'11928","reported_seq":12005,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.922998+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:02.922998+0000","last_peered":"2024-09-15T21:14:02.922998+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:02.922998+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6911244,"num_objects":844,"num_object_clones":0,"num_object_copies":3376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11926,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1522}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"35'11904","reported_seq":11980,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:02.947482+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:02.947482+0000","last_peered":"2024-09-15T21:14:02.947482+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:02.947482+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7017740,"num_objects":857,"num_object_clones":0,"num_object_copies":3428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1617}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11619","reported_seq":11686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.439008+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:05.439008+0000","last_peered":"2024-09-15T21:14:05.439008+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:05.439008+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6818328,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11756","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.180945+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:06.180945+0000","last_peered":"2024-09-15T21:14:06.180945+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:06.180945+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6599948,"num_objects":806,"num_object_clones":0,"num_object_copies":3224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11563","reported_seq":16232,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:05.490932+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:05.490932+0000","last_peered":"2024-09-15T21:14:05.490932+0000","last_clean":"2024-09-15T21:14:05.490932+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:05.490932+0000","last_undegraded":"2024-09-15T21:14:05.490932+0000","last_fullsized":"2024-09-15T21:14:05.490932+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6529024,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11593","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.473528+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:05.473528+0000","last_peered":"2024-09-15T21:14:05.473528+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:05.473528+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6830726,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"35'11707","reported_seq":16370,"reported_epoch":35,"state":"active+undersized+remapped","last_fresh":"2024-09-15T21:14:06.262176+0000","last_change":"2024-09-15T21:14:06.168049+0000","last_active":"2024-09-15T21:14:06.262176+0000","last_peered":"2024-09-15T21:14:06.262176+0000","last_clean":"2024-09-15T21:02:42.003716+0000","last_became_active":"2024-09-15T21:03:59.668598+0000","last_became_peered":"2024-09-15T21:03:59.668598+0000","last_unstale":"2024-09-15T21:14:06.262176+0000","last_undegraded":"2024-09-15T21:14:06.262176+0000","last_fullsized":"2024-09-15T21:03:59.040514+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6503046,"num_objects":794,"num_object_clones":0,"num_object_copies":3176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11706,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","1(0)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":794}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"35'11611","reported_seq":11671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.152534+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:05.152534+0000","last_peered":"2024-09-15T21:14:05.152534+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:05.152534+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722828,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11786","reported_seq":11875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.099305+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:06.099305+0000","last_peered":"2024-09-15T21:14:06.099305+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:06.099305+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7329036,"num_objects":895,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11784,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"35'11663","reported_seq":11738,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:03.205968+0000","last_change":"2024-09-15T21:03:59.662483+0000","last_active":"2024-09-15T21:14:03.205968+0000","last_peered":"2024-09-15T21:14:03.205968+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:03.205968+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7255308,"num_objects":886,"num_object_clones":0,"num_object_copies":3544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11661,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1429}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108839458,"num_objects":13291,"num_object_clones":0,"num_object_copies":53164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13291,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171787,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2309,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":218480640,"data_stored":218480640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160732,"ondisk_log_size":160732,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698448,"kb_used_data":53116,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93673392,"statfs":{"total":96636764160,"available":95921553408,"internally_reserved":0,"allocated":54390784,"data_stored":54217682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705836,"kb_used_data":52700,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93666004,"statfs":{"total":96636764160,"available":95913988096,"internally_reserved":0,"allocated":53964800,"data_stored":53791698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051705,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577180,"kb_used_data":54080,"kb_used_omap":4,"kb_used_meta":523067,"kb_avail":93794660,"statfs":{"total":96636764160,"available":96045731840,"internally_reserved":0,"allocated":55377920,"data_stored":55204818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":535621327},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,9,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247943,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":726816,"kb_used_data":55184,"kb_used_omap":4,"kb_used_meta":671611,"kb_avail":93645024,"statfs":{"total":96636764160,"available":95892504576,"internally_reserved":0,"allocated":56508416,"data_stored":56335314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":687730382},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,7,3,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55967744,"data_stored":55967744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":55062528,"data_stored":55062528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53424128,"data_stored":53424128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54026240,"data_stored":54026240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:11.239 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:11.861 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:11.861 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:11.895 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-15T21:14:10.186281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":106308130,"num_objects":12982,"num_object_clones":0,"num_object_copies":51928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172096,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160941,"ondisk_log_size":160941,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727124,"kb_used_data":207548,"kb_used_omap":14,"kb_used_meta":2519473,"kb_avail":374760236,"statfs":{"total":386547056640,"available":383754481664,"internally_reserved":0,"allocated":212529152,"data_stored":211820370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2579940914},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2,12,5,26,8,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10584064,"num_objects":-1292,"num_object_clones":0,"num_object_copies":-5168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1292,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006047"},"pg_stats":[{"pgid":"1.f","version":"35'11548","reported_seq":11600,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.324898+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:05.324898+0000","last_peered":"2024-09-15T21:14:05.324898+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:05.324898+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6586368,"num_objects":804,"num_object_clones":0,"num_object_copies":3216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1472}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11779","reported_seq":16677,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:09.110782+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:09.110782+0000","last_peered":"2024-09-15T21:14:09.110782+0000","last_clean":"2024-09-15T21:14:09.110782+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:09.110782+0000","last_undegraded":"2024-09-15T21:14:09.110782+0000","last_fullsized":"2024-09-15T21:14:09.110782+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6316032,"num_objects":771,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"35'11551","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.016100+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:06.016100+0000","last_peered":"2024-09-15T21:14:06.016100+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:06.016100+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7215774,"num_objects":882,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":882,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11550,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1376}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11686","reported_seq":11766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.031983+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:08.031983+0000","last_peered":"2024-09-15T21:14:08.031983+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:08.031983+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6305036,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11684,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"35'11846","reported_seq":11907,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.401565+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:05.401565+0000","last_peered":"2024-09-15T21:14:05.401565+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:05.401565+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6657292,"num_objects":813,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1498}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"35'11596","reported_seq":11647,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:04.966072+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:04.966072+0000","last_peered":"2024-09-15T21:14:04.966072+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:04.966072+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6389760,"num_objects":780,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11596,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11986","reported_seq":12067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.046941+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:08.046941+0000","last_peered":"2024-09-15T21:14:08.046941+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:08.046941+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436108,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11965","reported_seq":12045,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.233775+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:08.233775+0000","last_peered":"2024-09-15T21:14:08.233775+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:08.233775+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6518028,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1678}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"35'11619","reported_seq":11686,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.439008+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:05.439008+0000","last_peered":"2024-09-15T21:14:05.439008+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:05.439008+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6818328,"num_objects":833,"num_object_clones":0,"num_object_copies":3332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11616,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1499}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"35'11756","reported_seq":11844,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.180945+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:06.180945+0000","last_peered":"2024-09-15T21:14:06.180945+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:06.180945+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6599948,"num_objects":806,"num_object_clones":0,"num_object_copies":3224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11754,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1529}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"35'11563","reported_seq":16232,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-15T21:14:05.490932+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:05.490932+0000","last_peered":"2024-09-15T21:14:05.490932+0000","last_clean":"2024-09-15T21:14:05.490932+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:05.490932+0000","last_undegraded":"2024-09-15T21:14:05.490932+0000","last_fullsized":"2024-09-15T21:14:05.490932+0000","mapping_epoch":32,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6529024,"num_objects":797,"num_object_clones":0,"num_object_copies":3188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"35'11593","reported_seq":11638,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.473528+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:05.473528+0000","last_peered":"2024-09-15T21:14:05.473528+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:05.473528+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1500","ondisk_log_start":"27'1500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6830726,"num_objects":834,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11592,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1422}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11742","reported_seq":16430,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:14:09.514504+0000","last_change":"2024-09-15T21:14:08.443110+0000","last_active":"2024-09-15T21:14:09.514504+0000","last_peered":"2024-09-15T21:14:09.514504+0000","last_clean":"2024-09-15T21:14:09.514504+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:09.514504+0000","last_undegraded":"2024-09-15T21:14:09.514504+0000","last_fullsized":"2024-09-15T21:14:09.514504+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":748,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6216326,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11741,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"35'11611","reported_seq":11671,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:05.152534+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:05.152534+0000","last_peered":"2024-09-15T21:14:05.152534+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:05.152534+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6722828,"num_objects":821,"num_object_clones":0,"num_object_copies":3284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11609,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1458}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"35'11786","reported_seq":11875,"reported_epoch":35,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:06.099305+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:06.099305+0000","last_peered":"2024-09-15T21:14:06.099305+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:06.099305+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7329036,"num_objects":895,"num_object_clones":0,"num_object_copies":3580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":895,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11784,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11714","reported_seq":11858,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.219567+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:08.219567+0000","last_peered":"2024-09-15T21:14:08.219567+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:08.219567+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837516,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":106308130,"num_objects":12982,"num_object_clones":0,"num_object_copies":51928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172096,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":210784256,"data_stored":210784256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160941,"ondisk_log_size":160941,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698448,"kb_used_data":53116,"kb_used_omap":4,"kb_used_meta":645307,"kb_avail":93673392,"statfs":{"total":96636764160,"available":95921553408,"internally_reserved":0,"allocated":54390784,"data_stored":54217682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":660795088},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,3,1,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":705836,"kb_used_data":52700,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":93666004,"statfs":{"total":96636764160,"available":95913988096,"internally_reserved":0,"allocated":53964800,"data_stored":53791698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,10,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595708,"kb_used_data":50408,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93776132,"statfs":{"total":96636764160,"available":96026759168,"internally_reserved":0,"allocated":51617792,"data_stored":51436503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,10,0,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727132,"kb_used_data":51324,"kb_used_omap":4,"kb_used_meta":675771,"kb_avail":93644708,"statfs":{"total":96636764160,"available":95892180992,"internally_reserved":0,"allocated":52555776,"data_stored":52374487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":691990222},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52137984,"data_stored":52137984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51195904,"data_stored":51195904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53424128,"data_stored":53424128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54026240,"data_stored":54026240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:11.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:12.499 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:12.500 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:12.540 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-15T21:14:12.187069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101392930,"num_objects":12382,"num_object_clones":0,"num_object_copies":49528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172696,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2711512,"kb_used_data":200260,"kb_used_omap":14,"kb_used_meta":2511153,"kb_avail":374775848,"statfs":{"total":386547056640,"available":383770468352,"internally_reserved":0,"allocated":205066240,"data_stored":204341084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2571421234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,13,3,17,5,5,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13565952,"num_objects":-1656,"num_object_clones":0,"num_object_copies":-6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1656,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002982"},"pg_stats":[{"pgid":"1.f","version":"37'11604","reported_seq":11660,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.548446+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:10.548446+0000","last_peered":"2024-09-15T21:14:10.548446+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:10.548446+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6127616,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11779","reported_seq":16677,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:09.110782+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:09.110782+0000","last_peered":"2024-09-15T21:14:09.110782+0000","last_clean":"2024-09-15T21:14:09.110782+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:09.110782+0000","last_undegraded":"2024-09-15T21:14:09.110782+0000","last_fullsized":"2024-09-15T21:14:09.110782+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6316032,"num_objects":771,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11613","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.201477+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:11.201477+0000","last_peered":"2024-09-15T21:14:11.201477+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:11.201477+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6707870,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1438}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11686","reported_seq":11766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.031983+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:08.031983+0000","last_peered":"2024-09-15T21:14:08.031983+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:08.031983+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6305036,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11684,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11908","reported_seq":11975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.546716+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:10.546716+0000","last_peered":"2024-09-15T21:14:10.546716+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:10.546716+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6149388,"num_objects":751,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11654","reported_seq":11708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.579901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:10.579901+0000","last_peered":"2024-09-15T21:14:10.579901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:10.579901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5914624,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11986","reported_seq":12067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.046941+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:08.046941+0000","last_peered":"2024-09-15T21:14:08.046941+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:08.046941+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436108,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11965","reported_seq":12045,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.233775+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:08.233775+0000","last_peered":"2024-09-15T21:14:08.233775+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:08.233775+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6518028,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1678}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11673","reported_seq":11744,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.498914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:10.498914+0000","last_peered":"2024-09-15T21:14:10.498914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:10.498914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6375960,"num_objects":779,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11815","reported_seq":11907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.208989+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:11.208989+0000","last_peered":"2024-09-15T21:14:11.208989+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:11.208989+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6116620,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11620","reported_seq":16294,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:10.546420+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:10.546420+0000","last_peered":"2024-09-15T21:14:10.546420+0000","last_clean":"2024-09-15T21:14:10.546420+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:10.546420+0000","last_undegraded":"2024-09-15T21:14:10.546420+0000","last_fullsized":"2024-09-15T21:14:10.546420+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6062080,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11654","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.560278+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:10.560278+0000","last_peered":"2024-09-15T21:14:10.560278+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:10.560278+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6331014,"num_objects":773,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11653,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1483}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11742","reported_seq":16430,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:14:09.514504+0000","last_change":"2024-09-15T21:14:08.443110+0000","last_active":"2024-09-15T21:14:09.514504+0000","last_peered":"2024-09-15T21:14:09.514504+0000","last_clean":"2024-09-15T21:14:09.514504+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:09.514504+0000","last_undegraded":"2024-09-15T21:14:09.514504+0000","last_fullsized":"2024-09-15T21:14:09.514504+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":748,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6216326,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11741,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11665","reported_seq":11732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.450222+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:10.450222+0000","last_peered":"2024-09-15T21:14:10.450222+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:10.450222+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6280460,"num_objects":767,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11863","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.281114+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:11.281114+0000","last_peered":"2024-09-15T21:14:11.281114+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:11.281114+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698252,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11714","reported_seq":11858,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.219567+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:08.219567+0000","last_peered":"2024-09-15T21:14:08.219567+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:08.219567+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837516,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101392930,"num_objects":12382,"num_object_clones":0,"num_object_copies":49528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172696,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203218944,"data_stored":203218944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698968,"kb_used_data":49480,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93672872,"statfs":{"total":96636764160,"available":95921020928,"internally_reserved":0,"allocated":50667520,"data_stored":50486231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":49048,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":50225152,"data_stored":50043863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2,3,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595708,"kb_used_data":50408,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93776132,"statfs":{"total":96636764160,"available":96026759168,"internally_reserved":0,"allocated":51617792,"data_stored":51436503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,10,0,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727132,"kb_used_data":51324,"kb_used_omap":4,"kb_used_meta":675771,"kb_avail":93644708,"statfs":{"total":96636764160,"available":95892180992,"internally_reserved":0,"allocated":52555776,"data_stored":52374487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":691990222},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52137984,"data_stored":52137984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51195904,"data_stored":51195904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49635328,"data_stored":49635328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50249728,"data_stored":50249728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11604", 'reported_seq': 11660, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.548446+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:10.548446+0000', 'last_peered': '2024-09-15T21:14:10.548446+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:10.548446+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10004, 'log_dups_size': 0, 'ondisk_log_size': 10004, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6127616, 'num_objects': 748, 'num_object_clones': 0, 'num_object_copies': 2992, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 748, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11604, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11613", 'reported_seq': 11704, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:11.201477+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:11.201477+0000', 'last_peered': '2024-09-15T21:14:11.201477+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:11.201477+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10013, 'log_dups_size': 0, 'ondisk_log_size': 10013, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6707870, 'num_objects': 820, 'num_object_clones': 0, 'num_object_copies': 3280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 820, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11612, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1438}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11686", 'reported_seq': 11766, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:08.031983+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:08.031983+0000', 'last_peered': '2024-09-15T21:14:08.031983+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:08.031983+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6305036, 'num_objects': 770, 'num_object_clones': 0, 'num_object_copies': 3080, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 770, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11684, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1564}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11908", 'reported_seq': 11975, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.546716+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:10.546716+0000', 'last_peered': '2024-09-15T21:14:10.546716+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:10.546716+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6149388, 'num_objects': 751, 'num_object_clones': 0, 'num_object_copies': 3004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 751, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11906, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1560}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11654", 'reported_seq': 11708, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.579901+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:10.579901+0000', 'last_peered': '2024-09-15T21:14:10.579901+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:10.579901+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5914624, 'num_objects': 722, 'num_object_clones': 0, 'num_object_copies': 2888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11654, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1594}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'11986", 'reported_seq': 12067, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:08.046941+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:08.046941+0000', 'last_peered': '2024-09-15T21:14:08.046941+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:08.046941+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6436108, 'num_objects': 786, 'num_object_clones': 0, 'num_object_copies': 3144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 786, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11984, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1580}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'11965", 'reported_seq': 12045, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:08.233775+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:08.233775+0000', 'last_peered': '2024-09-15T21:14:08.233775+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:08.233775+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6518028, 'num_objects': 796, 'num_object_clones': 0, 'num_object_copies': 3184, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 796, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11963, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1678}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11673", 'reported_seq': 11744, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.498914+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:10.498914+0000', 'last_peered': '2024-09-15T21:14:10.498914+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:10.498914+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6375960, 'num_objects': 779, 'num_object_clones': 0, 'num_object_copies': 3116, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 779, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11670, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1553}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11815", 'reported_seq': 11907, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:11.208989+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:11.208989+0000', 'last_peered': '2024-09-15T21:14:11.208989+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:11.208989+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6116620, 'num_objects': 747, 'num_object_clones': 0, 'num_object_copies': 2988, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 747, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11813, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1588}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:12.543 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11654", 'reported_seq': 11704, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.560278+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:10.560278+0000', 'last_peered': '2024-09-15T21:14:10.560278+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:10.560278+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6331014, 'num_objects': 773, 'num_object_clones': 0, 'num_object_copies': 3092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 773, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11653, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1483}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.8 is not active+clean 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.8', 'version': "37'11742", 'reported_seq': 16430, 'reported_epoch': 37, 'state': 'active+clean+scrubbing', 'last_fresh': '2024-09-15T21:14:09.514504+0000', 'last_change': '2024-09-15T21:14:08.443110+0000', 'last_active': '2024-09-15T21:14:09.514504+0000', 'last_peered': '2024-09-15T21:14:09.514504+0000', 'last_clean': '2024-09-15T21:14:09.514504+0000', 'last_became_active': '2024-09-15T21:14:07.529588+0000', 'last_became_peered': '2024-09-15T21:14:07.529588+0000', 'last_unstale': '2024-09-15T21:14:09.514504+0000', 'last_undegraded': '2024-09-15T21:14:09.514504+0000', 'last_fullsized': '2024-09-15T21:14:09.514504+0000', 'mapping_epoch': 36, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 37, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 748, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'scrubbing for 1s', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6216326, 'num_objects': 759, 'num_object_clones': 0, 'num_object_copies': 3036, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 759, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11741, 'num_write_kb': 49999, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2288, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [1, 2, 3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11665", 'reported_seq': 11732, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:10.450222+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:10.450222+0000', 'last_peered': '2024-09-15T21:14:10.450222+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:10.450222+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6280460, 'num_objects': 767, 'num_object_clones': 0, 'num_object_copies': 3068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 767, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11663, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1512}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11863", 'reported_seq': 11957, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:11.281114+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:11.281114+0000', 'last_peered': '2024-09-15T21:14:11.281114+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:11.281114+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6698252, 'num_objects': 818, 'num_object_clones': 0, 'num_object_copies': 3272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 818, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11861, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1463}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:12.544 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11714", 'reported_seq': 11858, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:08.219567+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:08.219567+0000', 'last_peered': '2024-09-15T21:14:08.219567+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:08.219567+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6837516, 'num_objects': 835, 'num_object_clones': 0, 'num_object_copies': 3340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2285, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 835, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11712, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1450}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:12.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:13.316 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:13.316 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30122/49528 objects degraded (60.818%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":719,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":2},{"state_name":"active+clean+scrubbing","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12382,"data_bytes":101392930,"bytes_used":2776588288,"bytes_avail":383770468352,"bytes_total":386547056640,"degraded_objects":30122,"degraded_total":49528,"degraded_ratio":0.60818123081893072,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":206},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:13.341 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:13.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:13.970 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:13.970 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:13.994 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-09-15T21:14:12.187069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":101392930,"num_objects":12382,"num_object_clones":0,"num_object_copies":49528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172696,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2711512,"kb_used_data":200260,"kb_used_omap":14,"kb_used_meta":2511153,"kb_avail":374775848,"statfs":{"total":386547056640,"available":383770468352,"internally_reserved":0,"allocated":205066240,"data_stored":204341084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2571421234},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,13,3,17,5,5,2,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13565952,"num_objects":-1656,"num_object_clones":0,"num_object_copies":-6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-96,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1656,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002982"},"pg_stats":[{"pgid":"1.f","version":"37'11604","reported_seq":11660,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.548446+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:10.548446+0000","last_peered":"2024-09-15T21:14:10.548446+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:10.548446+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6127616,"num_objects":748,"num_object_clones":0,"num_object_copies":2992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11779","reported_seq":16677,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:09.110782+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:09.110782+0000","last_peered":"2024-09-15T21:14:09.110782+0000","last_clean":"2024-09-15T21:14:09.110782+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:09.110782+0000","last_undegraded":"2024-09-15T21:14:09.110782+0000","last_fullsized":"2024-09-15T21:14:09.110782+0000","mapping_epoch":34,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6316032,"num_objects":771,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2445,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11613","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.201477+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:11.201477+0000","last_peered":"2024-09-15T21:14:11.201477+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:11.201477+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6707870,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1438}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11686","reported_seq":11766,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.031983+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:08.031983+0000","last_peered":"2024-09-15T21:14:08.031983+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:08.031983+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6305036,"num_objects":770,"num_object_clones":0,"num_object_copies":3080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11684,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1564}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11908","reported_seq":11975,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.546716+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:10.546716+0000","last_peered":"2024-09-15T21:14:10.546716+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:10.546716+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6149388,"num_objects":751,"num_object_clones":0,"num_object_copies":3004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1560}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11654","reported_seq":11708,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.579901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:10.579901+0000","last_peered":"2024-09-15T21:14:10.579901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:10.579901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5914624,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11654,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1594}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'11986","reported_seq":12067,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.046941+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:08.046941+0000","last_peered":"2024-09-15T21:14:08.046941+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:08.046941+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6436108,"num_objects":786,"num_object_clones":0,"num_object_copies":3144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1580}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'11965","reported_seq":12045,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.233775+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:08.233775+0000","last_peered":"2024-09-15T21:14:08.233775+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:08.233775+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6518028,"num_objects":796,"num_object_clones":0,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1678}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11673","reported_seq":11744,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.498914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:10.498914+0000","last_peered":"2024-09-15T21:14:10.498914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:10.498914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6375960,"num_objects":779,"num_object_clones":0,"num_object_copies":3116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11670,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1553}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11815","reported_seq":11907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.208989+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:11.208989+0000","last_peered":"2024-09-15T21:14:11.208989+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:11.208989+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6116620,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11620","reported_seq":16294,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:10.546420+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:10.546420+0000","last_peered":"2024-09-15T21:14:10.546420+0000","last_clean":"2024-09-15T21:14:10.546420+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:10.546420+0000","last_undegraded":"2024-09-15T21:14:10.546420+0000","last_fullsized":"2024-09-15T21:14:10.546420+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6062080,"num_objects":740,"num_object_clones":0,"num_object_copies":2960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5731,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11654","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.560278+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:10.560278+0000","last_peered":"2024-09-15T21:14:10.560278+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:10.560278+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6331014,"num_objects":773,"num_object_clones":0,"num_object_copies":3092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11653,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1483}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11742","reported_seq":16430,"reported_epoch":37,"state":"active+clean+scrubbing","last_fresh":"2024-09-15T21:14:09.514504+0000","last_change":"2024-09-15T21:14:08.443110+0000","last_active":"2024-09-15T21:14:09.514504+0000","last_peered":"2024-09-15T21:14:09.514504+0000","last_clean":"2024-09-15T21:14:09.514504+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:09.514504+0000","last_undegraded":"2024-09-15T21:14:09.514504+0000","last_fullsized":"2024-09-15T21:14:09.514504+0000","mapping_epoch":36,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":748,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6216326,"num_objects":759,"num_object_clones":0,"num_object_copies":3036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11741,"num_write_kb":49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11665","reported_seq":11732,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:10.450222+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:10.450222+0000","last_peered":"2024-09-15T21:14:10.450222+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:10.450222+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6280460,"num_objects":767,"num_object_clones":0,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11663,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1512}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11863","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.281114+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:11.281114+0000","last_peered":"2024-09-15T21:14:11.281114+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:11.281114+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698252,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11714","reported_seq":11858,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:08.219567+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:08.219567+0000","last_peered":"2024-09-15T21:14:08.219567+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:08.219567+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6837516,"num_objects":835,"num_object_clones":0,"num_object_copies":3340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1450}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":101392930,"num_objects":12382,"num_object_clones":0,"num_object_copies":49528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12382,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172696,"num_write_kb":703907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2341,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":203218944,"data_stored":203218944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160741,"ondisk_log_size":160741,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698968,"kb_used_data":49480,"kb_used_omap":4,"kb_used_meta":649467,"kb_avail":93672872,"statfs":{"total":96636764160,"available":95921020928,"internally_reserved":0,"allocated":50667520,"data_stored":50486231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":665054928},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,1,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":49048,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":50225152,"data_stored":50043863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2,3,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051706,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595708,"kb_used_data":50408,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93776132,"statfs":{"total":96636764160,"available":96026759168,"internally_reserved":0,"allocated":51617792,"data_stored":51436503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,1,10,0,1,1,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247944,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727132,"kb_used_data":51324,"kb_used_omap":4,"kb_used_meta":675771,"kb_avail":93644708,"statfs":{"total":96636764160,"available":95892180992,"internally_reserved":0,"allocated":52555776,"data_stored":52374487,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":691990222},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52137984,"data_stored":52137984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51195904,"data_stored":51195904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49635328,"data_stored":49635328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50249728,"data_stored":50249728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:16.996 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:17.567 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:17.569 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:17.592 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":389,"stamp":"2024-09-15T21:14:16.188752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95127452,"num_objects":11617,"num_object_clones":0,"num_object_copies":46468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161723,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160605,"ondisk_log_size":160605,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2697144,"kb_used_data":188664,"kb_used_omap":14,"kb_used_meta":2508401,"kb_avail":374790216,"statfs":{"total":386547056640,"available":383785181184,"internally_reserved":0,"allocated":193191936,"data_stored":192466780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2568603186},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,10,8,9,9,4,5,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13712006,"num_objects":-1674,"num_object_clones":0,"num_object_copies":-6696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-10064,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.741961"},"pg_stats":[{"pgid":"1.f","version":"37'11668","reported_seq":11724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.740142+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:15.740142+0000","last_peered":"2024-09-15T21:14:15.740142+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:15.740142+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5603328,"num_objects":684,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11829","reported_seq":16727,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.457026+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:14.457026+0000","last_peered":"2024-09-15T21:14:14.457026+0000","last_clean":"2024-09-15T21:14:14.457026+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:14.457026+0000","last_undegraded":"2024-09-15T21:14:14.457026+0000","last_fullsized":"2024-09-15T21:14:14.457026+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906432,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11613","reported_seq":11704,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.201477+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:11.201477+0000","last_peered":"2024-09-15T21:14:11.201477+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:11.201477+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6707870,"num_objects":820,"num_object_clones":0,"num_object_copies":3280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":820,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11612,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1438}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11736","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.205299+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:13.205299+0000","last_peered":"2024-09-15T21:14:13.205299+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:13.205299+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895436,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11734,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11964","reported_seq":12031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.399654+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:15.399654+0000","last_peered":"2024-09-15T21:14:15.399654+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:15.399654+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5690636,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11707","reported_seq":11762,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.331788+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:15.331788+0000","last_peered":"2024-09-15T21:14:15.331788+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:15.331788+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12053","reported_seq":12134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.268508+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:13.268508+0000","last_peered":"2024-09-15T21:14:13.268508+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:13.268508+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887244,"num_objects":719,"num_object_clones":0,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12019","reported_seq":12099,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.094880+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:13.094880+0000","last_peered":"2024-09-15T21:14:13.094880+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:13.094880+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.557850+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:15.557850+0000","last_peered":"2024-09-15T21:14:15.557850+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:15.557850+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5859864,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11815","reported_seq":11907,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.208989+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:11.208989+0000","last_peered":"2024-09-15T21:14:11.208989+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:11.208989+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6116620,"num_objects":747,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11813,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1588}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11684","reported_seq":16358,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:15.634842+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:15.634842+0000","last_peered":"2024-09-15T21:14:15.634842+0000","last_clean":"2024-09-15T21:14:15.634842+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:15.634842+0000","last_undegraded":"2024-09-15T21:14:15.634842+0000","last_fullsized":"2024-09-15T21:14:15.634842+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5537792,"num_objects":676,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11705","reported_seq":11756,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.517834+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:15.517834+0000","last_peered":"2024-09-15T21:14:15.517834+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:15.517834+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913222,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11805","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.603395+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:14.603395+0000","last_peered":"2024-09-15T21:14:14.603395+0000","last_clean":"2024-09-15T21:14:14.603395+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:14.603395+0000","last_undegraded":"2024-09-15T21:14:14.603395+0000","last_fullsized":"2024-09-15T21:14:14.603395+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11728","reported_seq":11794,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.627023+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:15.627023+0000","last_peered":"2024-09-15T21:14:15.627023+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:15.627023+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1575}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11863","reported_seq":11957,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:11.281114+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:11.281114+0000","last_peered":"2024-09-15T21:14:11.281114+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:11.281114+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6698252,"num_objects":818,"num_object_clones":0,"num_object_copies":3272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1463}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11780","reported_seq":12013,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.298542+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:13.298542+0000","last_peered":"2024-09-15T21:14:13.298542+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:13.298542+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6296844,"num_objects":769,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95127452,"num_objects":11617,"num_object_clones":0,"num_object_copies":46468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11617,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161723,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":191320064,"data_stored":191320064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160605,"ondisk_log_size":160605,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717268,"kb_used_data":45572,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93654572,"statfs":{"total":96636764160,"available":95902281728,"internally_reserved":0,"allocated":46665728,"data_stored":46484439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,2,3,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":689704,"kb_used_data":49048,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":93682136,"statfs":{"total":96636764160,"available":95930507264,"internally_reserved":0,"allocated":50225152,"data_stored":50043863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,2,3,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579296,"kb_used_data":46480,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93792544,"statfs":{"total":96636764160,"available":96043565056,"internally_reserved":0,"allocated":47595520,"data_stored":47414231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710876,"kb_used_data":47564,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93660964,"statfs":{"total":96636764160,"available":95908827136,"internally_reserved":0,"allocated":48705536,"data_stored":48524247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48218112,"data_stored":48218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47255552,"data_stored":47255552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49635328,"data_stored":49635328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46211072,"data_stored":46211072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:17.593 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:18.228 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:18.228 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:18.261 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-15T21:14:18.189674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93497244,"num_objects":11418,"num_object_clones":0,"num_object_copies":45672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161922,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2715620,"kb_used_data":184932,"kb_used_omap":14,"kb_used_meta":2530609,"kb_avail":374771740,"statfs":{"total":386547056640,"available":383766261760,"internally_reserved":0,"allocated":189370368,"data_stored":188645212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2591344178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,11,7,15,13,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15342214,"num_objects":-1873,"num_object_clones":0,"num_object_copies":-7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9865,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.726260"},"pg_stats":[{"pgid":"1.f","version":"37'11668","reported_seq":11724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.740142+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:15.740142+0000","last_peered":"2024-09-15T21:14:15.740142+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:15.740142+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5603328,"num_objects":684,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11829","reported_seq":16727,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.457026+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:14.457026+0000","last_peered":"2024-09-15T21:14:14.457026+0000","last_clean":"2024-09-15T21:14:14.457026+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:14.457026+0000","last_undegraded":"2024-09-15T21:14:14.457026+0000","last_fullsized":"2024-09-15T21:14:14.457026+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906432,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11690","reported_seq":11781,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.105174+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:16.105174+0000","last_peered":"2024-09-15T21:14:16.105174+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:16.105174+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6077086,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11689,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11736","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.205299+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:13.205299+0000","last_peered":"2024-09-15T21:14:13.205299+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:13.205299+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895436,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11734,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11964","reported_seq":12031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.399654+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:15.399654+0000","last_peered":"2024-09-15T21:14:15.399654+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:15.399654+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5690636,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11707","reported_seq":11762,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.331788+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:15.331788+0000","last_peered":"2024-09-15T21:14:15.331788+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:15.331788+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12053","reported_seq":12134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.268508+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:13.268508+0000","last_peered":"2024-09-15T21:14:13.268508+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:13.268508+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887244,"num_objects":719,"num_object_clones":0,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12019","reported_seq":12099,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.094880+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:13.094880+0000","last_peered":"2024-09-15T21:14:13.094880+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:13.094880+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.557850+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:15.557850+0000","last_peered":"2024-09-15T21:14:15.557850+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:15.557850+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5859864,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11877","reported_seq":11969,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.264091+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:16.264091+0000","last_peered":"2024-09-15T21:14:16.264091+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:16.264091+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5608716,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11875,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11684","reported_seq":16358,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:15.634842+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:15.634842+0000","last_peered":"2024-09-15T21:14:15.634842+0000","last_clean":"2024-09-15T21:14:15.634842+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:15.634842+0000","last_undegraded":"2024-09-15T21:14:15.634842+0000","last_fullsized":"2024-09-15T21:14:15.634842+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5537792,"num_objects":676,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11705","reported_seq":11756,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.517834+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:15.517834+0000","last_peered":"2024-09-15T21:14:15.517834+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:15.517834+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913222,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11805","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.603395+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:14.603395+0000","last_peered":"2024-09-15T21:14:14.603395+0000","last_clean":"2024-09-15T21:14:14.603395+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:14.603395+0000","last_undegraded":"2024-09-15T21:14:14.603395+0000","last_fullsized":"2024-09-15T21:14:14.603395+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11728","reported_seq":11794,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.627023+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:15.627023+0000","last_peered":"2024-09-15T21:14:15.627023+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:15.627023+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1575}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11923","reported_seq":12017,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.416881+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:16.416881+0000","last_peered":"2024-09-15T21:14:16.416881+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:16.416881+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6206732,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11780","reported_seq":12013,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.298542+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:13.298542+0000","last_peered":"2024-09-15T21:14:13.298542+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:13.298542+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6296844,"num_objects":769,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":93497244,"num_objects":11418,"num_object_clones":0,"num_object_copies":45672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161922,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187326464,"data_stored":187326464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717268,"kb_used_data":45572,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93654572,"statfs":{"total":96636764160,"available":95902281728,"internally_reserved":0,"allocated":46665728,"data_stored":46484439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,2,3,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708180,"kb_used_data":45316,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93663660,"statfs":{"total":96636764160,"available":95911587840,"internally_reserved":0,"allocated":46403584,"data_stored":46222295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579296,"kb_used_data":46480,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93792544,"statfs":{"total":96636764160,"available":96043565056,"internally_reserved":0,"allocated":47595520,"data_stored":47414231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710876,"kb_used_data":47564,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93660964,"statfs":{"total":96636764160,"available":95908827136,"internally_reserved":0,"allocated":48705536,"data_stored":48524247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48218112,"data_stored":48218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47255552,"data_stored":47255552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45641728,"data_stored":45641728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46211072,"data_stored":46211072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:18.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:18.262 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11668", 'reported_seq': 11724, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.740142+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:15.740142+0000', 'last_peered': '2024-09-15T21:14:15.740142+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:15.740142+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10068, 'log_dups_size': 0, 'ondisk_log_size': 10068, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5603328, 'num_objects': 684, 'num_object_clones': 0, 'num_object_copies': 2736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 684, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11668, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1592}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.262 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11690", 'reported_seq': 11781, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:16.105174+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:16.105174+0000', 'last_peered': '2024-09-15T21:14:16.105174+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:16.105174+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1600", 'ondisk_log_start': "27'1600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10090, 'log_dups_size': 0, 'ondisk_log_size': 10090, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6077086, 'num_objects': 743, 'num_object_clones': 0, 'num_object_copies': 2972, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 743, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11689, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1515}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11736", 'reported_seq': 11816, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:13.205299+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:13.205299+0000', 'last_peered': '2024-09-15T21:14:13.205299+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:13.205299+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5895436, 'num_objects': 720, 'num_object_clones': 0, 'num_object_copies': 2880, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 720, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11734, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1614}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'11964", 'reported_seq': 12031, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.399654+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:15.399654+0000', 'last_peered': '2024-09-15T21:14:15.399654+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:15.399654+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5690636, 'num_objects': 695, 'num_object_clones': 0, 'num_object_copies': 2780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 695, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11962, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11707", 'reported_seq': 11762, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.331788+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:15.331788+0000', 'last_peered': '2024-09-15T21:14:15.331788+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:15.331788+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10007, 'log_dups_size': 0, 'ondisk_log_size': 10007, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5480448, 'num_objects': 669, 'num_object_clones': 0, 'num_object_copies': 2676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11707, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1647}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12053", 'reported_seq': 12134, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:13.268508+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:13.268508+0000', 'last_peered': '2024-09-15T21:14:13.268508+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:13.268508+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5887244, 'num_objects': 719, 'num_object_clones': 0, 'num_object_copies': 2876, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 719, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12051, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1647}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:18.263 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12019", 'reported_seq': 12099, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:13.094880+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:13.094880+0000', 'last_peered': '2024-09-15T21:14:13.094880+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:13.094880+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10019, 'log_dups_size': 0, 'ondisk_log_size': 10019, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6075660, 'num_objects': 742, 'num_object_clones': 0, 'num_object_copies': 2968, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 742, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12017, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1732}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11736", 'reported_seq': 11807, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.557850+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:15.557850+0000', 'last_peered': '2024-09-15T21:14:15.557850+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:15.557850+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5859864, 'num_objects': 716, 'num_object_clones': 0, 'num_object_copies': 2864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 716, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11733, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1616}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11877", 'reported_seq': 11969, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:16.264091+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:16.264091+0000', 'last_peered': '2024-09-15T21:14:16.264091+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:16.264091+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5608716, 'num_objects': 685, 'num_object_clones': 0, 'num_object_copies': 2740, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 685, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11875, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1650}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11705", 'reported_seq': 11756, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.517834+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:15.517834+0000', 'last_peered': '2024-09-15T21:14:15.517834+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:15.517834+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5913222, 'num_objects': 722, 'num_object_clones': 0, 'num_object_copies': 2888, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 722, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11704, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1534}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11728", 'reported_seq': 11794, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:15.627023+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:15.627023+0000', 'last_peered': '2024-09-15T21:14:15.627023+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:15.627023+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5764364, 'num_objects': 704, 'num_object_clones': 0, 'num_object_copies': 2816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11726, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1575}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:18.264 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11923", 'reported_seq': 12017, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:16.416881+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:16.416881+0000', 'last_peered': '2024-09-15T21:14:16.416881+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:16.416881+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10023, 'log_dups_size': 0, 'ondisk_log_size': 10023, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6206732, 'num_objects': 758, 'num_object_clones': 0, 'num_object_copies': 3032, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 758, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11921, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1523}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:18.265 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:18.265 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11780", 'reported_seq': 12013, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:13.298542+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:13.298542+0000', 'last_peered': '2024-09-15T21:14:13.298542+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:13.298542+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10080, 'log_dups_size': 0, 'ondisk_log_size': 10080, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6296844, 'num_objects': 769, 'num_object_clones': 0, 'num_object_copies': 3076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 769, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11778, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 78, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:18.265 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:19.011 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:19.012 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30078/45672 objects degraded (65.857%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":724,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":11418,"data_bytes":93497244,"bytes_used":2780794880,"bytes_avail":383766261760,"bytes_total":386547056640,"degraded_objects":30078,"degraded_total":45672,"degraded_ratio":0.65856542301629006},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:19.038 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:14:19.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:19.618 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:19.618 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:19.646 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-15T21:14:18.189674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93497244,"num_objects":11418,"num_object_clones":0,"num_object_copies":45672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161922,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2715620,"kb_used_data":184932,"kb_used_omap":14,"kb_used_meta":2530609,"kb_avail":374771740,"statfs":{"total":386547056640,"available":383766261760,"internally_reserved":0,"allocated":189370368,"data_stored":188645212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2591344178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,11,7,15,13,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15342214,"num_objects":-1873,"num_object_clones":0,"num_object_copies":-7492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9865,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.726260"},"pg_stats":[{"pgid":"1.f","version":"37'11668","reported_seq":11724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.740142+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:15.740142+0000","last_peered":"2024-09-15T21:14:15.740142+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:15.740142+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5603328,"num_objects":684,"num_object_clones":0,"num_object_copies":2736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11668,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1592}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11829","reported_seq":16727,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.457026+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:14.457026+0000","last_peered":"2024-09-15T21:14:14.457026+0000","last_clean":"2024-09-15T21:14:14.457026+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:14.457026+0000","last_undegraded":"2024-09-15T21:14:14.457026+0000","last_fullsized":"2024-09-15T21:14:14.457026+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5906432,"num_objects":721,"num_object_clones":0,"num_object_copies":2884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11690","reported_seq":11781,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.105174+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:16.105174+0000","last_peered":"2024-09-15T21:14:16.105174+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:16.105174+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6077086,"num_objects":743,"num_object_clones":0,"num_object_copies":2972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11689,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1515}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11736","reported_seq":11816,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.205299+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:13.205299+0000","last_peered":"2024-09-15T21:14:13.205299+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:13.205299+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5895436,"num_objects":720,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11734,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1614}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'11964","reported_seq":12031,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.399654+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:15.399654+0000","last_peered":"2024-09-15T21:14:15.399654+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:15.399654+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5690636,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11962,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11707","reported_seq":11762,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.331788+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:15.331788+0000","last_peered":"2024-09-15T21:14:15.331788+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:15.331788+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5480448,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11707,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12053","reported_seq":12134,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.268508+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:13.268508+0000","last_peered":"2024-09-15T21:14:13.268508+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:13.268508+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5887244,"num_objects":719,"num_object_clones":0,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1647}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12019","reported_seq":12099,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.094880+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:13.094880+0000","last_peered":"2024-09-15T21:14:13.094880+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:13.094880+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6075660,"num_objects":742,"num_object_clones":0,"num_object_copies":2968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1732}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11736","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.557850+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:15.557850+0000","last_peered":"2024-09-15T21:14:15.557850+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:15.557850+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5859864,"num_objects":716,"num_object_clones":0,"num_object_copies":2864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11733,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1616}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11877","reported_seq":11969,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.264091+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:16.264091+0000","last_peered":"2024-09-15T21:14:16.264091+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:16.264091+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5608716,"num_objects":685,"num_object_clones":0,"num_object_copies":2740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11875,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11684","reported_seq":16358,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:15.634842+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:15.634842+0000","last_peered":"2024-09-15T21:14:15.634842+0000","last_clean":"2024-09-15T21:14:15.634842+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:15.634842+0000","last_undegraded":"2024-09-15T21:14:15.634842+0000","last_fullsized":"2024-09-15T21:14:15.634842+0000","mapping_epoch":32,"log_start":"27'1600","ondisk_log_start":"27'1600","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5537792,"num_objects":676,"num_object_clones":0,"num_object_copies":2704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11705","reported_seq":11756,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.517834+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:15.517834+0000","last_peered":"2024-09-15T21:14:15.517834+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:15.517834+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5913222,"num_objects":722,"num_object_clones":0,"num_object_copies":2888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11704,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1534}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11805","reported_seq":16499,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:14.603395+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:14.603395+0000","last_peered":"2024-09-15T21:14:14.603395+0000","last_clean":"2024-09-15T21:14:14.603395+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:14.603395+0000","last_undegraded":"2024-09-15T21:14:14.603395+0000","last_fullsized":"2024-09-15T21:14:14.603395+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5693440,"num_objects":695,"num_object_clones":0,"num_object_copies":2780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11728","reported_seq":11794,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:15.627023+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:15.627023+0000","last_peered":"2024-09-15T21:14:15.627023+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:15.627023+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11726,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1575}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11923","reported_seq":12017,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:16.416881+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:16.416881+0000","last_peered":"2024-09-15T21:14:16.416881+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:16.416881+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6206732,"num_objects":758,"num_object_clones":0,"num_object_copies":3032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11921,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1523}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11780","reported_seq":12013,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:13.298542+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:13.298542+0000","last_peered":"2024-09-15T21:14:13.298542+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:13.298542+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6296844,"num_objects":769,"num_object_clones":0,"num_object_copies":3076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11778,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":93497244,"num_objects":11418,"num_object_clones":0,"num_object_copies":45672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30078,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11418,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":161922,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":187326464,"data_stored":187326464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160704,"ondisk_log_size":160704,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717268,"kb_used_data":45572,"kb_used_omap":4,"kb_used_meta":671675,"kb_avail":93654572,"statfs":{"total":96636764160,"available":95902281728,"internally_reserved":0,"allocated":46665728,"data_stored":46484439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":687795920},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,5,2,3,5,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708180,"kb_used_data":45316,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93663660,"statfs":{"total":96636764160,"available":95911587840,"internally_reserved":0,"allocated":46403584,"data_stored":46222295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051707,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579296,"kb_used_data":46480,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93792544,"statfs":{"total":96636764160,"available":96043565056,"internally_reserved":0,"allocated":47595520,"data_stored":47414231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247945,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710876,"kb_used_data":47564,"kb_used_omap":4,"kb_used_meta":663291,"kb_avail":93660964,"statfs":{"total":96636764160,"available":95908827136,"internally_reserved":0,"allocated":48705536,"data_stored":48524247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":679210702},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48218112,"data_stored":48218112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47255552,"data_stored":47255552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45641728,"data_stored":45641728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46211072,"data_stored":46211072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:22.648 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:23.305 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:23.305 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:23.340 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-15T21:14:22.191295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85567388,"num_objects":10450,"num_object_clones":0,"num_object_copies":41800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718468,"kb_used_data":169720,"kb_used_omap":14,"kb_used_meta":2548657,"kb_avail":374768892,"statfs":{"total":386547056640,"available":383763345408,"internally_reserved":0,"allocated":173793280,"data_stored":173068124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2609825330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,5,3,9,14,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20740742,"num_objects":-2532,"num_object_clones":0,"num_object_copies":-10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9206,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005014"},"pg_stats":[{"pgid":"1.f","version":"37'11721","reported_seq":11777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.768129+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:20.768129+0000","last_peered":"2024-09-15T21:14:20.768129+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:20.768129+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11721,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11893","reported_seq":16791,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.574077+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:19.574077+0000","last_peered":"2024-09-15T21:14:19.574077+0000","last_clean":"2024-09-15T21:14:19.574077+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:19.574077+0000","last_undegraded":"2024-09-15T21:14:19.574077+0000","last_fullsized":"2024-09-15T21:14:19.574077+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5382144,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11758","reported_seq":11849,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.272011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:21.272011+0000","last_peered":"2024-09-15T21:14:21.272011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:21.272011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5520030,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11757,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11798","reported_seq":11878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.090573+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:18.090573+0000","last_peered":"2024-09-15T21:14:18.090573+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:18.090573+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5387532,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12020","reported_seq":12087,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.669656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:20.669656+0000","last_peered":"2024-09-15T21:14:20.669656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:20.669656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5231884,"num_objects":639,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11752","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.523128+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:20.523128+0000","last_peered":"2024-09-15T21:14:20.523128+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:20.523128+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11752,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12103","reported_seq":12184,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.381717+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:18.381717+0000","last_peered":"2024-09-15T21:14:18.381717+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:18.381717+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5477644,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12073","reported_seq":12153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.159958+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:18.159958+0000","last_peered":"2024-09-15T21:14:18.159958+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:18.159958+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5633292,"num_objects":688,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11808","reported_seq":11879,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.705291+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:20.705291+0000","last_peered":"2024-09-15T21:14:20.705291+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:20.705291+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5270040,"num_objects":644,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11805,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11936","reported_seq":12028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.965595+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:20.965595+0000","last_peered":"2024-09-15T21:14:20.965595+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:20.965595+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5125388,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11934,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":16425,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:20.699107+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:20.699107+0000","last_peered":"2024-09-15T21:14:20.699107+0000","last_clean":"2024-09-15T21:14:20.699107+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:20.699107+0000","last_undegraded":"2024-09-15T21:14:20.699107+0000","last_fullsized":"2024-09-15T21:14:20.699107+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11763","reported_seq":11814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.765281+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:20.765281+0000","last_peered":"2024-09-15T21:14:20.765281+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:20.765281+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5438086,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11762,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11869","reported_seq":16563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.240832+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:19.240832+0000","last_peered":"2024-09-15T21:14:19.240832+0000","last_clean":"2024-09-15T21:14:19.240832+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:19.240832+0000","last_undegraded":"2024-09-15T21:14:19.240832+0000","last_fullsized":"2024-09-15T21:14:19.240832+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11796","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.770012+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:20.770012+0000","last_peered":"2024-09-15T21:14:20.770012+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:20.770012+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5207308,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11977","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.350899+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:21.350899+0000","last_peered":"2024-09-15T21:14:21.350899+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:21.350899+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11853","reported_seq":12167,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.506413+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:18.506413+0000","last_peered":"2024-09-15T21:14:18.506413+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:18.506413+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698828,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":85567388,"num_objects":10450,"num_object_clones":0,"num_object_copies":41800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171442176,"data_stored":171442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701060,"kb_used_data":41840,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93670780,"statfs":{"total":96636764160,"available":95918878720,"internally_reserved":0,"allocated":42844160,"data_stored":42662871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708552,"kb_used_data":41520,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93663288,"statfs":{"total":96636764160,"available":95911206912,"internally_reserved":0,"allocated":42516480,"data_stored":42335191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579664,"kb_used_data":42688,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93792176,"statfs":{"total":96636764160,"available":96043188224,"internally_reserved":0,"allocated":43712512,"data_stored":43531223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729192,"kb_used_data":43672,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93642648,"statfs":{"total":96636764160,"available":95890071552,"internally_reserved":0,"allocated":44720128,"data_stored":44538839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,2,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44085248,"data_stored":44085248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43323392,"data_stored":43323392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41746432,"data_stored":41746432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":42287104,"data_stored":42287104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:23.341 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:24.005 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:24.006 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:24.033 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-15T21:14:22.191295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":85567388,"num_objects":10450,"num_object_clones":0,"num_object_copies":41800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2718468,"kb_used_data":169720,"kb_used_omap":14,"kb_used_meta":2548657,"kb_avail":374768892,"statfs":{"total":386547056640,"available":383763345408,"internally_reserved":0,"allocated":173793280,"data_stored":173068124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2609825330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,5,3,9,14,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20740742,"num_objects":-2532,"num_object_clones":0,"num_object_copies":-10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-85,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9206,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2203,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005014"},"pg_stats":[{"pgid":"1.f","version":"37'11721","reported_seq":11777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.768129+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:20.768129+0000","last_peered":"2024-09-15T21:14:20.768129+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:20.768129+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11721,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11893","reported_seq":16791,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.574077+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:19.574077+0000","last_peered":"2024-09-15T21:14:19.574077+0000","last_clean":"2024-09-15T21:14:19.574077+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:19.574077+0000","last_undegraded":"2024-09-15T21:14:19.574077+0000","last_fullsized":"2024-09-15T21:14:19.574077+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5382144,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11758","reported_seq":11849,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.272011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:21.272011+0000","last_peered":"2024-09-15T21:14:21.272011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:21.272011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5520030,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11757,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11798","reported_seq":11878,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.090573+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:18.090573+0000","last_peered":"2024-09-15T21:14:18.090573+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:18.090573+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5387532,"num_objects":658,"num_object_clones":0,"num_object_copies":2632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11796,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1676}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12020","reported_seq":12087,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.669656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:20.669656+0000","last_peered":"2024-09-15T21:14:20.669656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:20.669656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5231884,"num_objects":639,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11752","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.523128+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:20.523128+0000","last_peered":"2024-09-15T21:14:20.523128+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:20.523128+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11752,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12103","reported_seq":12184,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.381717+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:18.381717+0000","last_peered":"2024-09-15T21:14:18.381717+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:18.381717+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5477644,"num_objects":669,"num_object_clones":0,"num_object_copies":2676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12101,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12073","reported_seq":12153,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.159958+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:18.159958+0000","last_peered":"2024-09-15T21:14:18.159958+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:18.159958+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5633292,"num_objects":688,"num_object_clones":0,"num_object_copies":2752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12071,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1786}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11808","reported_seq":11879,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.705291+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:20.705291+0000","last_peered":"2024-09-15T21:14:20.705291+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:20.705291+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5270040,"num_objects":644,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11805,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11936","reported_seq":12028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.965595+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:20.965595+0000","last_peered":"2024-09-15T21:14:20.965595+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:20.965595+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5125388,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11934,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":16425,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:20.699107+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:20.699107+0000","last_peered":"2024-09-15T21:14:20.699107+0000","last_clean":"2024-09-15T21:14:20.699107+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:20.699107+0000","last_undegraded":"2024-09-15T21:14:20.699107+0000","last_fullsized":"2024-09-15T21:14:20.699107+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11763","reported_seq":11814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.765281+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:20.765281+0000","last_peered":"2024-09-15T21:14:20.765281+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:20.765281+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5438086,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11762,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11869","reported_seq":16563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.240832+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:19.240832+0000","last_peered":"2024-09-15T21:14:19.240832+0000","last_clean":"2024-09-15T21:14:19.240832+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:19.240832+0000","last_undegraded":"2024-09-15T21:14:19.240832+0000","last_fullsized":"2024-09-15T21:14:19.240832+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11796","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.770012+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:20.770012+0000","last_peered":"2024-09-15T21:14:20.770012+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:20.770012+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5207308,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11977","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.350899+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:21.350899+0000","last_peered":"2024-09-15T21:14:21.350899+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:21.350899+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11853","reported_seq":12167,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:18.506413+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:18.506413+0000","last_peered":"2024-09-15T21:14:18.506413+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:18.506413+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5698828,"num_objects":696,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11851,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1504}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":85567388,"num_objects":10450,"num_object_clones":0,"num_object_copies":41800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30037,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10450,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":162890,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":171442176,"data_stored":171442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160872,"ondisk_log_size":160872,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701060,"kb_used_data":41840,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93670780,"statfs":{"total":96636764160,"available":95918878720,"internally_reserved":0,"allocated":42844160,"data_stored":42662871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708552,"kb_used_data":41520,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93663288,"statfs":{"total":96636764160,"available":95911206912,"internally_reserved":0,"allocated":42516480,"data_stored":42335191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579664,"kb_used_data":42688,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93792176,"statfs":{"total":96636764160,"available":96043188224,"internally_reserved":0,"allocated":43712512,"data_stored":43531223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247946,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729192,"kb_used_data":43672,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93642648,"statfs":{"total":96636764160,"available":95890071552,"internally_reserved":0,"allocated":44720128,"data_stored":44538839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,7,1,2,0,4,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":44085248,"data_stored":44085248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43323392,"data_stored":43323392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41746432,"data_stored":41746432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":42287104,"data_stored":42287104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11721", 'reported_seq': 11777, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.768129+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:20.768129+0000', 'last_peered': '2024-09-15T21:14:20.768129+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:20.768129+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5169152, 'num_objects': 631, 'num_object_clones': 0, 'num_object_copies': 2524, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 631, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11721, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1645}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11758", 'reported_seq': 11849, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:21.272011+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:21.272011+0000', 'last_peered': '2024-09-15T21:14:21.272011+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:21.272011+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10058, 'log_dups_size': 0, 'ondisk_log_size': 10058, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5520030, 'num_objects': 675, 'num_object_clones': 0, 'num_object_copies': 2700, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 675, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11757, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1583}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11798", 'reported_seq': 11878, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:18.090573+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:18.090573+0000', 'last_peered': '2024-09-15T21:14:18.090573+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:18.090573+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5387532, 'num_objects': 658, 'num_object_clones': 0, 'num_object_copies': 2632, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 658, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11796, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1676}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12020", 'reported_seq': 12087, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.669656+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:20.669656+0000', 'last_peered': '2024-09-15T21:14:20.669656+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:20.669656+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5231884, 'num_objects': 639, 'num_object_clones': 0, 'num_object_copies': 2556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 639, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12018, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1672}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.035 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11752", 'reported_seq': 11807, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.523128+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:20.523128+0000', 'last_peered': '2024-09-15T21:14:20.523128+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:20.523128+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5111808, 'num_objects': 624, 'num_object_clones': 0, 'num_object_copies': 2496, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 624, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11752, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1692}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12103", 'reported_seq': 12184, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:18.381717+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:18.381717+0000', 'last_peered': '2024-09-15T21:14:18.381717+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:18.381717+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5477644, 'num_objects': 669, 'num_object_clones': 0, 'num_object_copies': 2676, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 669, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12101, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1697}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12073", 'reported_seq': 12153, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:18.159958+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:18.159958+0000', 'last_peered': '2024-09-15T21:14:18.159958+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:18.159958+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10073, 'log_dups_size': 0, 'ondisk_log_size': 10073, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5633292, 'num_objects': 688, 'num_object_clones': 0, 'num_object_copies': 2752, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 688, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12071, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1786}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11808", 'reported_seq': 11879, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.705291+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:20.705291+0000', 'last_peered': '2024-09-15T21:14:20.705291+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:20.705291+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5270040, 'num_objects': 644, 'num_object_clones': 0, 'num_object_copies': 2576, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 644, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11805, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1688}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'11936", 'reported_seq': 12028, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.965595+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:20.965595+0000', 'last_peered': '2024-09-15T21:14:20.965595+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:20.965595+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5125388, 'num_objects': 626, 'num_object_clones': 0, 'num_object_copies': 2504, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 626, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11934, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1709}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:24.036 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11763", 'reported_seq': 11814, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.765281+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:20.765281+0000', 'last_peered': '2024-09-15T21:14:20.765281+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:20.765281+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5438086, 'num_objects': 664, 'num_object_clones': 0, 'num_object_copies': 2656, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 664, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11762, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11796", 'reported_seq': 11864, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:20.770012+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:20.770012+0000', 'last_peered': '2024-09-15T21:14:20.770012+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:20.770012+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5207308, 'num_objects': 636, 'num_object_clones': 0, 'num_object_copies': 2544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11794, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1643}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'11977", 'reported_seq': 12071, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:21.350899+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:21.350899+0000', 'last_peered': '2024-09-15T21:14:21.350899+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:21.350899+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10077, 'log_dups_size': 0, 'ondisk_log_size': 10077, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5764364, 'num_objects': 704, 'num_object_clones': 0, 'num_object_copies': 2816, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 704, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11975, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1577}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:24.037 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11853", 'reported_seq': 12167, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:18.506413+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:18.506413+0000', 'last_peered': '2024-09-15T21:14:18.506413+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:18.506413+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5698828, 'num_objects': 696, 'num_object_clones': 0, 'num_object_copies': 2784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 696, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11851, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 119, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1504}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:24.037 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:24.759 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:24.763 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30001/40804 objects degraded (73.525%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":730,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10201,"data_bytes":83527580,"bytes_used":2779648000,"bytes_avail":383767408640,"bytes_total":386547056640,"degraded_objects":30001,"degraded_total":40804,"degraded_ratio":0.73524654445642579},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:24.789 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:14:24.789 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:25.395 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:25.396 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:25.420 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-15T21:14:24.191822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":83527580,"num_objects":10201,"num_object_clones":0,"num_object_copies":40804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163139,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2714500,"kb_used_data":165748,"kb_used_omap":14,"kb_used_meta":2548657,"kb_avail":374772860,"statfs":{"total":386547056640,"available":383767408640,"internally_reserved":0,"allocated":169725952,"data_stored":169000796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2609825330},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,8,1,9,13,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17865350,"num_objects":-2181,"num_object_clones":0,"num_object_copies":-8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-9557,"num_write_kb":-49999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2167,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004753"},"pg_stats":[{"pgid":"1.f","version":"37'11721","reported_seq":11777,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.768129+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:20.768129+0000","last_peered":"2024-09-15T21:14:20.768129+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:20.768129+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11721,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1645}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11893","reported_seq":16791,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.574077+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:19.574077+0000","last_peered":"2024-09-15T21:14:19.574077+0000","last_clean":"2024-09-15T21:14:19.574077+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:19.574077+0000","last_undegraded":"2024-09-15T21:14:19.574077+0000","last_fullsized":"2024-09-15T21:14:19.574077+0000","mapping_epoch":34,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5382144,"num_objects":657,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2559,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11758","reported_seq":11849,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.272011+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:21.272011+0000","last_peered":"2024-09-15T21:14:21.272011+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:21.272011+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5520030,"num_objects":675,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":675,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11757,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1583}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11861","reported_seq":11940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.385503+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:23.385503+0000","last_peered":"2024-09-15T21:14:23.385503+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:23.385503+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4871436,"num_objects":595,"num_object_clones":0,"num_object_copies":2380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12020","reported_seq":12087,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.669656+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:20.669656+0000","last_peered":"2024-09-15T21:14:20.669656+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:20.669656+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5231884,"num_objects":639,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1672}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11752","reported_seq":11807,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.523128+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:20.523128+0000","last_peered":"2024-09-15T21:14:20.523128+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:20.523128+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111808,"num_objects":624,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11752,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12172","reported_seq":12253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.268756+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:23.268756+0000","last_peered":"2024-09-15T21:14:23.268756+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:23.268756+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912396,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12145","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.383924+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:23.383924+0000","last_peered":"2024-09-15T21:14:23.383924+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:23.383924+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5043468,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11808","reported_seq":11879,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.705291+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:20.705291+0000","last_peered":"2024-09-15T21:14:20.705291+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:20.705291+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5270040,"num_objects":644,"num_object_clones":0,"num_object_copies":2576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11805,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1688}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'11936","reported_seq":12028,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.965595+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:20.965595+0000","last_peered":"2024-09-15T21:14:20.965595+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:20.965595+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5125388,"num_objects":626,"num_object_clones":0,"num_object_copies":2504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11934,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1709}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11752","reported_seq":16425,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:20.699107+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:20.699107+0000","last_peered":"2024-09-15T21:14:20.699107+0000","last_clean":"2024-09-15T21:14:20.699107+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:20.699107+0000","last_undegraded":"2024-09-15T21:14:20.699107+0000","last_fullsized":"2024-09-15T21:14:20.699107+0000","mapping_epoch":32,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":608,"num_object_clones":0,"num_object_copies":2432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5863,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11763","reported_seq":11814,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.765281+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:20.765281+0000","last_peered":"2024-09-15T21:14:20.765281+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:20.765281+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5438086,"num_objects":664,"num_object_clones":0,"num_object_copies":2656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11762,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11869","reported_seq":16563,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:19.240832+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:19.240832+0000","last_peered":"2024-09-15T21:14:19.240832+0000","last_clean":"2024-09-15T21:14:19.240832+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:19.240832+0000","last_undegraded":"2024-09-15T21:14:19.240832+0000","last_fullsized":"2024-09-15T21:14:19.240832+0000","mapping_epoch":36,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5169152,"num_objects":631,"num_object_clones":0,"num_object_copies":2524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11796","reported_seq":11864,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:20.770012+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:20.770012+0000","last_peered":"2024-09-15T21:14:20.770012+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:20.770012+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5207308,"num_objects":636,"num_object_clones":0,"num_object_copies":2544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11794,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1643}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'11977","reported_seq":12071,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:21.350899+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:21.350899+0000","last_peered":"2024-09-15T21:14:21.350899+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:21.350899+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5764364,"num_objects":704,"num_object_clones":0,"num_object_copies":2816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1577}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11898","reported_seq":12285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.387712+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:23.387712+0000","last_peered":"2024-09-15T21:14:23.387712+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:23.387712+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5330188,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11896,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":83527580,"num_objects":10201,"num_object_clones":0,"num_object_copies":40804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10201,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163139,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167489536,"data_stored":167489536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160921,"ondisk_log_size":160921,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701060,"kb_used_data":41840,"kb_used_omap":4,"kb_used_meta":659195,"kb_avail":93670780,"statfs":{"total":96636764160,"available":95918878720,"internally_reserved":0,"allocated":42844160,"data_stored":42662871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":675016400},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708552,"kb_used_data":41520,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":93663288,"statfs":{"total":96636764160,"available":95911206912,"internally_reserved":0,"allocated":42516480,"data_stored":42335191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051708,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579664,"kb_used_data":42688,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93792176,"statfs":{"total":96636764160,"available":96043188224,"internally_reserved":0,"allocated":43712512,"data_stored":43531223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725224,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93646616,"statfs":{"total":96636764160,"available":95894134784,"internally_reserved":0,"allocated":40652800,"data_stored":40471511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,0,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40132608,"data_stored":40132608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43323392,"data_stored":43323392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":41746432,"data_stored":41746432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":42287104,"data_stored":42287104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:28.422 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:29.027 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:29.027 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:29.065 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-15T21:14:28.193169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":77670300,"num_objects":9486,"num_object_clones":0,"num_object_copies":37944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163854,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733744,"kb_used_data":154472,"kb_used_omap":14,"kb_used_meta":2579185,"kb_avail":374753616,"statfs":{"total":386547056640,"available":383747702784,"internally_reserved":0,"allocated":158179328,"data_stored":157454172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2641086002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,7,8,6,10,12,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17457152,"num_objects":-2131,"num_object_clones":0,"num_object_copies":-8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004416"},"pg_stats":[{"pgid":"1.f","version":"37'11775","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.743152+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:25.743152+0000","last_peered":"2024-09-15T21:14:25.743152+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:25.743152+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11941","reported_seq":16839,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:24.396305+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:24.396305+0000","last_peered":"2024-09-15T21:14:24.396305+0000","last_clean":"2024-09-15T21:14:24.396305+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:24.396305+0000","last_undegraded":"2024-09-15T21:14:24.396305+0000","last_fullsized":"2024-09-15T21:14:24.396305+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4988928,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11805","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.302368+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:26.302368+0000","last_peered":"2024-09-15T21:14:26.302368+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:26.302368+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5135006,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11804,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11861","reported_seq":11940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.385503+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:23.385503+0000","last_peered":"2024-09-15T21:14:23.385503+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:23.385503+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4871436,"num_objects":595,"num_object_clones":0,"num_object_copies":2380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12089","reported_seq":12154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.806942+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:25.806942+0000","last_peered":"2024-09-15T21:14:25.806942+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:25.806942+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666636,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11810","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.312061+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:25.312061+0000","last_peered":"2024-09-15T21:14:25.312061+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:25.312061+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4636672,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12172","reported_seq":12253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.268756+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:23.268756+0000","last_peered":"2024-09-15T21:14:23.268756+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:23.268756+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912396,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12145","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.383924+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:23.383924+0000","last_peered":"2024-09-15T21:14:23.383924+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:23.383924+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5043468,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11863","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.786670+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:25.786670+0000","last_peered":"2024-09-15T21:14:25.786670+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:25.786670+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4819480,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12003","reported_seq":12095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.278134+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:26.278134+0000","last_peered":"2024-09-15T21:14:26.278134+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:26.278134+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11812","reported_seq":16486,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:25.413990+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:25.413990+0000","last_peered":"2024-09-15T21:14:25.413990+0000","last_clean":"2024-09-15T21:14:25.413990+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:25.413990+0000","last_undegraded":"2024-09-15T21:14:25.413990+0000","last_fullsized":"2024-09-15T21:14:25.413990+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4489216,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.530003+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:25.530003+0000","last_peered":"2024-09-15T21:14:25.530003+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:25.530003+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4881030,"num_objects":596,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11925","reported_seq":16619,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:24.365117+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:24.365117+0000","last_peered":"2024-09-15T21:14:24.365117+0000","last_clean":"2024-09-15T21:14:24.365117+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:24.365117+0000","last_undegraded":"2024-09-15T21:14:24.365117+0000","last_fullsized":"2024-09-15T21:14:24.365117+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4710400,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11857","reported_seq":11925,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.760993+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:25.760993+0000","last_peered":"2024-09-15T21:14:25.760993+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:25.760993+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707596,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1704}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12049","reported_seq":12141,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.362888+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:26.362888+0000","last_peered":"2024-09-15T21:14:26.362888+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:26.362888+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5174540,"num_objects":632,"num_object_clones":0,"num_object_copies":2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11898","reported_seq":12285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.387712+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:23.387712+0000","last_peered":"2024-09-15T21:14:23.387712+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:23.387712+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5330188,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11896,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":77670300,"num_objects":9486,"num_object_clones":0,"num_object_copies":37944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163854,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155664384,"data_stored":155664384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701516,"kb_used_data":38136,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93670324,"statfs":{"total":96636764160,"available":95918411776,"internally_reserved":0,"allocated":39051264,"data_stored":38869975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727020,"kb_used_data":37788,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93644820,"statfs":{"total":96636764160,"available":95892295680,"internally_reserved":0,"allocated":38694912,"data_stored":38513623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579984,"kb_used_data":38848,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93791856,"statfs":{"total":96636764160,"available":96042860544,"internally_reserved":0,"allocated":39780352,"data_stored":39599063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725224,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93646616,"statfs":{"total":96636764160,"available":95894134784,"internally_reserved":0,"allocated":40652800,"data_stored":40471511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,0,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40132608,"data_stored":40132608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39313408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37875712,"data_stored":37875712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38342656,"data_stored":38342656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:29.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:29.708 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:29.709 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:29.729 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-15T21:14:28.193169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":77670300,"num_objects":9486,"num_object_clones":0,"num_object_copies":37944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163854,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2733744,"kb_used_data":154472,"kb_used_omap":14,"kb_used_meta":2579185,"kb_avail":374753616,"statfs":{"total":386547056640,"available":383747702784,"internally_reserved":0,"allocated":158179328,"data_stored":157454172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2641086002},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,7,8,6,10,12,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17457152,"num_objects":-2131,"num_object_clones":0,"num_object_copies":-8524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2131,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004416"},"pg_stats":[{"pgid":"1.f","version":"37'11775","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.743152+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:25.743152+0000","last_peered":"2024-09-15T21:14:25.743152+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:25.743152+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'11941","reported_seq":16839,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:24.396305+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:24.396305+0000","last_peered":"2024-09-15T21:14:24.396305+0000","last_clean":"2024-09-15T21:14:24.396305+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:24.396305+0000","last_undegraded":"2024-09-15T21:14:24.396305+0000","last_fullsized":"2024-09-15T21:14:24.396305+0000","mapping_epoch":34,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4988928,"num_objects":609,"num_object_clones":0,"num_object_copies":2436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2607,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11805","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.302368+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:26.302368+0000","last_peered":"2024-09-15T21:14:26.302368+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:26.302368+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5135006,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11804,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11861","reported_seq":11940,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.385503+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:23.385503+0000","last_peered":"2024-09-15T21:14:23.385503+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:23.385503+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4871436,"num_objects":595,"num_object_clones":0,"num_object_copies":2380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11859,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1739}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12089","reported_seq":12154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.806942+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:25.806942+0000","last_peered":"2024-09-15T21:14:25.806942+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:25.806942+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666636,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11810","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.312061+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:25.312061+0000","last_peered":"2024-09-15T21:14:25.312061+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:25.312061+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4636672,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12172","reported_seq":12253,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.268756+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:23.268756+0000","last_peered":"2024-09-15T21:14:23.268756+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:23.268756+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4912396,"num_objects":600,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12170,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1766}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12145","reported_seq":12225,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.383924+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:23.383924+0000","last_peered":"2024-09-15T21:14:23.383924+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:23.383924+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5043468,"num_objects":616,"num_object_clones":0,"num_object_copies":2464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12143,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1858}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11863","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.786670+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:25.786670+0000","last_peered":"2024-09-15T21:14:25.786670+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:25.786670+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4819480,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12003","reported_seq":12095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.278134+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:26.278134+0000","last_peered":"2024-09-15T21:14:26.278134+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:26.278134+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11812","reported_seq":16486,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:25.413990+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:25.413990+0000","last_peered":"2024-09-15T21:14:25.413990+0000","last_clean":"2024-09-15T21:14:25.413990+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:25.413990+0000","last_undegraded":"2024-09-15T21:14:25.413990+0000","last_fullsized":"2024-09-15T21:14:25.413990+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4489216,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.530003+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:25.530003+0000","last_peered":"2024-09-15T21:14:25.530003+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:25.530003+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4881030,"num_objects":596,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11925","reported_seq":16619,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:24.365117+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:24.365117+0000","last_peered":"2024-09-15T21:14:24.365117+0000","last_clean":"2024-09-15T21:14:24.365117+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:24.365117+0000","last_undegraded":"2024-09-15T21:14:24.365117+0000","last_fullsized":"2024-09-15T21:14:24.365117+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4710400,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11857","reported_seq":11925,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.760993+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:25.760993+0000","last_peered":"2024-09-15T21:14:25.760993+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:25.760993+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707596,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1704}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12049","reported_seq":12141,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.362888+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:26.362888+0000","last_peered":"2024-09-15T21:14:26.362888+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:26.362888+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5174540,"num_objects":632,"num_object_clones":0,"num_object_copies":2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11898","reported_seq":12285,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:23.387712+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:23.387712+0000","last_peered":"2024-09-15T21:14:23.387712+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:23.387712+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5330188,"num_objects":651,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11896,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1513}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":77670300,"num_objects":9486,"num_object_clones":0,"num_object_copies":37944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9486,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":163854,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155664384,"data_stored":155664384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160736,"ondisk_log_size":160736,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701516,"kb_used_data":38136,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93670324,"statfs":{"total":96636764160,"available":95918411776,"internally_reserved":0,"allocated":39051264,"data_stored":38869975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727020,"kb_used_data":37788,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93644820,"statfs":{"total":96636764160,"available":95892295680,"internally_reserved":0,"allocated":38694912,"data_stored":38513623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051709,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":579984,"kb_used_data":38848,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93791856,"statfs":{"total":96636764160,"available":96042860544,"internally_reserved":0,"allocated":39780352,"data_stored":39599063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,1,2,3,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":725224,"kb_used_data":39700,"kb_used_omap":4,"kb_used_meta":685499,"kb_avail":93646616,"statfs":{"total":96636764160,"available":95894134784,"internally_reserved":0,"allocated":40652800,"data_stored":40471511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":701951694},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,0,0,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40132608,"data_stored":40132608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":39313408,"data_stored":39313408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37875712,"data_stored":37875712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38342656,"data_stored":38342656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11775", 'reported_seq': 11831, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.743152+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:25.743152+0000', 'last_peered': '2024-09-15T21:14:25.743152+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:25.743152+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1700", 'ondisk_log_start': "27'1700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4726784, 'num_objects': 577, 'num_object_clones': 0, 'num_object_copies': 2308, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 577, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11775, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1699}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11805", 'reported_seq': 11896, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:26.302368+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:26.302368+0000', 'last_peered': '2024-09-15T21:14:26.302368+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:26.302368+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10005, 'log_dups_size': 0, 'ondisk_log_size': 10005, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5135006, 'num_objects': 628, 'num_object_clones': 0, 'num_object_copies': 2512, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 628, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11804, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1630}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11861", 'reported_seq': 11940, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:23.385503+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:23.385503+0000', 'last_peered': '2024-09-15T21:14:23.385503+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:23.385503+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4871436, 'num_objects': 595, 'num_object_clones': 0, 'num_object_copies': 2380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 595, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11859, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1739}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12089", 'reported_seq': 12154, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.806942+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:25.806942+0000', 'last_peered': '2024-09-15T21:14:25.806942+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:25.806942+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4666636, 'num_objects': 570, 'num_object_clones': 0, 'num_object_copies': 2280, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 570, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12087, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1741}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:29.731 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11810", 'reported_seq': 11865, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.312061+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:25.312061+0000', 'last_peered': '2024-09-15T21:14:25.312061+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:25.312061+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4636672, 'num_objects': 566, 'num_object_clones': 0, 'num_object_copies': 2264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 566, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11810, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1750}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12172", 'reported_seq': 12253, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:23.268756+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:23.268756+0000', 'last_peered': '2024-09-15T21:14:23.268756+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:23.268756+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4912396, 'num_objects': 600, 'num_object_clones': 0, 'num_object_copies': 2400, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 600, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12170, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1766}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12145", 'reported_seq': 12225, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:23.383924+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:23.383924+0000', 'last_peered': '2024-09-15T21:14:23.383924+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:23.383924+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10045, 'log_dups_size': 0, 'ondisk_log_size': 10045, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5043468, 'num_objects': 616, 'num_object_clones': 0, 'num_object_copies': 2464, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 616, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12143, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1858}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11863", 'reported_seq': 11934, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.786670+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:25.786670+0000', 'last_peered': '2024-09-15T21:14:25.786670+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:25.786670+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10063, 'log_dups_size': 0, 'ondisk_log_size': 10063, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4819480, 'num_objects': 589, 'num_object_clones': 0, 'num_object_copies': 2356, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 589, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11860, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1743}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12003", 'reported_seq': 12095, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:26.278134+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:26.278134+0000', 'last_peered': '2024-09-15T21:14:26.278134+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:26.278134+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4576524, 'num_objects': 559, 'num_object_clones': 0, 'num_object_copies': 2236, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 559, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12001, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1776}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:29.732 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11831", 'reported_seq': 11882, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.530003+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:25.530003+0000', 'last_peered': '2024-09-15T21:14:25.530003+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:25.530003+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10031, 'log_dups_size': 0, 'ondisk_log_size': 10031, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4881030, 'num_objects': 596, 'num_object_clones': 0, 'num_object_copies': 2384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 596, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11830, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1660}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11857", 'reported_seq': 11925, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:25.760993+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:25.760993+0000', 'last_peered': '2024-09-15T21:14:25.760993+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:25.760993+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4707596, 'num_objects': 575, 'num_object_clones': 0, 'num_object_copies': 2300, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 575, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11855, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1704}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12049", 'reported_seq': 12141, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:26.362888+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:26.362888+0000', 'last_peered': '2024-09-15T21:14:26.362888+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:26.362888+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10049, 'log_dups_size': 0, 'ondisk_log_size': 10049, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5174540, 'num_objects': 632, 'num_object_clones': 0, 'num_object_copies': 2528, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 632, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12047, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1649}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:29.733 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'11898", 'reported_seq': 12285, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:23.387712+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:23.387712+0000', 'last_peered': '2024-09-15T21:14:23.387712+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:23.387712+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10098, 'log_dups_size': 0, 'ondisk_log_size': 10098, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5330188, 'num_objects': 651, 'num_object_clones': 0, 'num_object_copies': 2604, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2164, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 651, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11896, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 155, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1513}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:29.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:30.451 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:30.451 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30001/37944 objects degraded (79.067%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":9486,"data_bytes":77670300,"bytes_used":2799353856,"bytes_avail":383747702784,"bytes_total":386547056640,"degraded_objects":30001,"degraded_total":37944,"degraded_ratio":0.79066519080750586,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":177},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:30.472 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:30.473 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:31.078 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:31.110 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-15T21:14:30.193811+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":74803100,"num_objects":9136,"num_object_clones":0,"num_object_copies":36544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164204,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160686,"ondisk_log_size":160686,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2752272,"kb_used_data":146636,"kb_used_omap":14,"kb_used_meta":2605553,"kb_avail":374735088,"statfs":{"total":386547056640,"available":383728730112,"internally_reserved":0,"allocated":150155264,"data_stored":149430108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2668086834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,9,13,13,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18694144,"num_objects":-2282,"num_object_clones":0,"num_object_copies":-9128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2282,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004136"},"pg_stats":[{"pgid":"1.f","version":"37'11775","reported_seq":11831,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.743152+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:25.743152+0000","last_peered":"2024-09-15T21:14:25.743152+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:25.743152+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1700","ondisk_log_start":"27'1700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4726784,"num_objects":577,"num_object_clones":0,"num_object_copies":2308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12015","reported_seq":16912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.677381+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:29.677381+0000","last_peered":"2024-09-15T21:14:29.677381+0000","last_clean":"2024-09-15T21:14:29.677381+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:29.677381+0000","last_undegraded":"2024-09-15T21:14:29.677381+0000","last_fullsized":"2024-09-15T21:14:29.677381+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4382720,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11805","reported_seq":11896,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.302368+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:26.302368+0000","last_peered":"2024-09-15T21:14:26.302368+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:26.302368+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5135006,"num_objects":628,"num_object_clones":0,"num_object_copies":2512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11804,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1630}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11914","reported_seq":11994,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:28.435799+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:28.435799+0000","last_peered":"2024-09-15T21:14:28.435799+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:28.435799+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4437260,"num_objects":542,"num_object_clones":0,"num_object_copies":2168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":542,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1792}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12089","reported_seq":12154,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.806942+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:25.806942+0000","last_peered":"2024-09-15T21:14:25.806942+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:25.806942+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4666636,"num_objects":570,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1741}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11810","reported_seq":11865,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.312061+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:25.312061+0000","last_peered":"2024-09-15T21:14:25.312061+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:25.312061+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4636672,"num_objects":566,"num_object_clones":0,"num_object_copies":2264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1750}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12226","reported_seq":12307,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:28.437016+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:28.437016+0000","last_peered":"2024-09-15T21:14:28.437016+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:28.437016+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4470028,"num_objects":546,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1820}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12194","reported_seq":12274,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:28.080449+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:28.080449+0000","last_peered":"2024-09-15T21:14:28.080449+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:28.080449+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4642060,"num_objects":567,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12192,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1907}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11863","reported_seq":11934,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.786670+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:25.786670+0000","last_peered":"2024-09-15T21:14:25.786670+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:25.786670+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4819480,"num_objects":589,"num_object_clones":0,"num_object_copies":2356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11860,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12003","reported_seq":12095,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.278134+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:26.278134+0000","last_peered":"2024-09-15T21:14:26.278134+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:26.278134+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4576524,"num_objects":559,"num_object_clones":0,"num_object_copies":2236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1776}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11812","reported_seq":16486,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:25.413990+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:25.413990+0000","last_peered":"2024-09-15T21:14:25.413990+0000","last_clean":"2024-09-15T21:14:25.413990+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:25.413990+0000","last_undegraded":"2024-09-15T21:14:25.413990+0000","last_fullsized":"2024-09-15T21:14:25.413990+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4489216,"num_objects":548,"num_object_clones":0,"num_object_copies":2192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5923,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11831","reported_seq":11882,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.530003+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:25.530003+0000","last_peered":"2024-09-15T21:14:25.530003+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:25.530003+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4881030,"num_objects":596,"num_object_clones":0,"num_object_copies":2384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11830,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1660}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11978","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.704439+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:29.704439+0000","last_peered":"2024-09-15T21:14:29.704439+0000","last_clean":"2024-09-15T21:14:29.704439+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:29.704439+0000","last_undegraded":"2024-09-15T21:14:29.704439+0000","last_fullsized":"2024-09-15T21:14:29.704439+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11857","reported_seq":11925,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:25.760993+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:25.760993+0000","last_peered":"2024-09-15T21:14:25.760993+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:25.760993+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4707596,"num_objects":575,"num_object_clones":0,"num_object_copies":2300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":575,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1704}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12049","reported_seq":12141,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:26.362888+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:26.362888+0000","last_peered":"2024-09-15T21:14:26.362888+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:26.362888+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5174540,"num_objects":632,"num_object_clones":0,"num_object_copies":2528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12047,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1649}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'11965","reported_seq":12436,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:28.436514+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:28.436514+0000","last_peered":"2024-09-15T21:14:28.436514+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:28.436514+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4781324,"num_objects":584,"num_object_clones":0,"num_object_copies":2336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11963,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":197,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1538}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":74803100,"num_objects":9136,"num_object_clones":0,"num_object_copies":36544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9136,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164204,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":216,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147681280,"data_stored":147681280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160686,"ondisk_log_size":160686,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":701516,"kb_used_data":38136,"kb_used_omap":4,"kb_used_meta":663355,"kb_avail":93670324,"statfs":{"total":96636764160,"available":95918411776,"internally_reserved":0,"allocated":39051264,"data_stored":38869975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":679276240},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,4,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727020,"kb_used_data":37788,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93644820,"statfs":{"total":96636764160,"available":95892295680,"internally_reserved":0,"allocated":38694912,"data_stored":38513623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580248,"kb_used_data":34952,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93791592,"statfs":{"total":96636764160,"available":96042590208,"internally_reserved":0,"allocated":35790848,"data_stored":35609559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247948,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":743488,"kb_used_data":35760,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93628352,"statfs":{"total":96636764160,"available":95875432448,"internally_reserved":0,"allocated":36618240,"data_stored":36436951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":724692686},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4,5,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36114432,"data_stored":36114432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35348480,"data_stored":35348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":37875712,"data_stored":37875712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":38342656,"data_stored":38342656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:34.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:34.668 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:34.668 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:34.697 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-15T21:14:34.195078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67848092,"num_objects":8287,"num_object_clones":0,"num_object_copies":33148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165053,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754612,"kb_used_data":135092,"kb_used_omap":14,"kb_used_meta":2619441,"kb_avail":374732748,"statfs":{"total":386547056640,"available":383726333952,"internally_reserved":0,"allocated":138334208,"data_stored":137609052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2682308146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,9,11,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17719296,"num_objects":-2163,"num_object_clones":0,"num_object_copies":-8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003782"},"pg_stats":[{"pgid":"1.f","version":"37'11842","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.774538+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:30.774538+0000","last_peered":"2024-09-15T21:14:30.774538+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:30.774538+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11842,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12015","reported_seq":16912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.677381+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:29.677381+0000","last_peered":"2024-09-15T21:14:29.677381+0000","last_clean":"2024-09-15T21:14:29.677381+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:29.677381+0000","last_undegraded":"2024-09-15T21:14:29.677381+0000","last_fullsized":"2024-09-15T21:14:29.677381+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4382720,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11871","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.333026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:31.333026+0000","last_peered":"2024-09-15T21:14:31.333026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:31.333026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594334,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11870,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11966","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.259598+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:33.259598+0000","last_peered":"2024-09-15T21:14:33.259598+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:33.259598+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011276,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12165","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.739809+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:30.739809+0000","last_peered":"2024-09-15T21:14:30.739809+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:30.739809+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4044044,"num_objects":494,"num_object_clones":0,"num_object_copies":1976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11861","reported_seq":11917,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.832854+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:30.832854+0000","last_peered":"2024-09-15T21:14:30.832854+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:30.832854+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4218880,"num_objects":515,"num_object_clones":0,"num_object_copies":2060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12286","reported_seq":12364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.562373+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:33.562373+0000","last_peered":"2024-09-15T21:14:33.562373+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:33.562373+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978508,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12284,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12262","reported_seq":12342,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.404734+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:33.404734+0000","last_peered":"2024-09-15T21:14:33.404734+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:33.404734+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4085004,"num_objects":499,"num_object_clones":0,"num_object_copies":1996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11917","reported_seq":11988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.826669+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:30.826669+0000","last_peered":"2024-09-15T21:14:30.826669+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:30.826669+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4377112,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12074","reported_seq":12166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.469804+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:31.469804+0000","last_peered":"2024-09-15T21:14:31.469804+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:31.469804+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3994892,"num_objects":488,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1847}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11863","reported_seq":16537,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:30.823523+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:30.823523+0000","last_peered":"2024-09-15T21:14:30.823523+0000","last_clean":"2024-09-15T21:14:30.823523+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:30.823523+0000","last_undegraded":"2024-09-15T21:14:30.823523+0000","last_fullsized":"2024-09-15T21:14:30.823523+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4071424,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11896","reported_seq":11947,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.514079+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:30.514079+0000","last_peered":"2024-09-15T21:14:30.514079+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:30.514079+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348550,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11978","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.704439+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:29.704439+0000","last_peered":"2024-09-15T21:14:29.704439+0000","last_clean":"2024-09-15T21:14:29.704439+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:29.704439+0000","last_undegraded":"2024-09-15T21:14:29.704439+0000","last_fullsized":"2024-09-15T21:14:29.704439+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11915","reported_seq":11983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.717192+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:30.717192+0000","last_peered":"2024-09-15T21:14:30.717192+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:30.717192+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4232460,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1762}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12100","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.430821+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:31.430821+0000","last_peered":"2024-09-15T21:14:31.430821+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:31.430821+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756748,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12024","reported_seq":12563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.544247+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:33.544247+0000","last_peered":"2024-09-15T21:14:33.544247+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:33.544247+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297996,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67848092,"num_objects":8287,"num_object_clones":0,"num_object_copies":33148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165053,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135970816,"data_stored":135970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719828,"kb_used_data":34244,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93652012,"statfs":{"total":96636764160,"available":95899660288,"internally_reserved":0,"allocated":35065856,"data_stored":34884567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727340,"kb_used_data":33948,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93644500,"statfs":{"total":96636764160,"available":95891968000,"internally_reserved":0,"allocated":34762752,"data_stored":34581463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580248,"kb_used_data":34952,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93791592,"statfs":{"total":96636764160,"available":96042590208,"internally_reserved":0,"allocated":35790848,"data_stored":35609559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727196,"kb_used_data":31948,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93644644,"statfs":{"total":96636764160,"available":95892115456,"internally_reserved":0,"allocated":32714752,"data_stored":32533463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32243712,"data_stored":32243712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35348480,"data_stored":35348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33935360,"data_stored":33935360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34443264,"data_stored":34443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:34.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:35.390 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:35.390 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:35.413 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-15T21:14:34.195078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67848092,"num_objects":8287,"num_object_clones":0,"num_object_copies":33148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165053,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2754612,"kb_used_data":135092,"kb_used_omap":14,"kb_used_meta":2619441,"kb_avail":374732748,"statfs":{"total":386547056640,"available":383726333952,"internally_reserved":0,"allocated":138334208,"data_stored":137609052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2682308146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,8,9,11,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17719296,"num_objects":-2163,"num_object_clones":0,"num_object_copies":-8652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2163,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003782"},"pg_stats":[{"pgid":"1.f","version":"37'11842","reported_seq":11895,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.774538+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:30.774538+0000","last_peered":"2024-09-15T21:14:30.774538+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:30.774538+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4177920,"num_objects":510,"num_object_clones":0,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11842,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1766}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12015","reported_seq":16912,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.677381+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:29.677381+0000","last_peered":"2024-09-15T21:14:29.677381+0000","last_clean":"2024-09-15T21:14:29.677381+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:29.677381+0000","last_undegraded":"2024-09-15T21:14:29.677381+0000","last_fullsized":"2024-09-15T21:14:29.677381+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4382720,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11871","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.333026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:31.333026+0000","last_peered":"2024-09-15T21:14:31.333026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:31.333026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594334,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11870,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11966","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.259598+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:33.259598+0000","last_peered":"2024-09-15T21:14:33.259598+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:33.259598+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011276,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12165","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.739809+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:30.739809+0000","last_peered":"2024-09-15T21:14:30.739809+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:30.739809+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4044044,"num_objects":494,"num_object_clones":0,"num_object_copies":1976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1817}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11861","reported_seq":11917,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.832854+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:30.832854+0000","last_peered":"2024-09-15T21:14:30.832854+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:30.832854+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4218880,"num_objects":515,"num_object_clones":0,"num_object_copies":2060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11861,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1801}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12286","reported_seq":12364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.562373+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:33.562373+0000","last_peered":"2024-09-15T21:14:33.562373+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:33.562373+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978508,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12284,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12262","reported_seq":12342,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.404734+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:33.404734+0000","last_peered":"2024-09-15T21:14:33.404734+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:33.404734+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4085004,"num_objects":499,"num_object_clones":0,"num_object_copies":1996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11917","reported_seq":11988,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.826669+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:30.826669+0000","last_peered":"2024-09-15T21:14:30.826669+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:30.826669+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4377112,"num_objects":535,"num_object_clones":0,"num_object_copies":2140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11914,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1797}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12074","reported_seq":12166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.469804+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:31.469804+0000","last_peered":"2024-09-15T21:14:31.469804+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:31.469804+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3994892,"num_objects":488,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1847}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11863","reported_seq":16537,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:30.823523+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:30.823523+0000","last_peered":"2024-09-15T21:14:30.823523+0000","last_clean":"2024-09-15T21:14:30.823523+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:30.823523+0000","last_undegraded":"2024-09-15T21:14:30.823523+0000","last_fullsized":"2024-09-15T21:14:30.823523+0000","mapping_epoch":32,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4071424,"num_objects":497,"num_object_clones":0,"num_object_copies":1988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11896","reported_seq":11947,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.514079+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:30.514079+0000","last_peered":"2024-09-15T21:14:30.514079+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:30.514079+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4348550,"num_objects":531,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11895,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1725}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'11978","reported_seq":16672,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:29.704439+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:29.704439+0000","last_peered":"2024-09-15T21:14:29.704439+0000","last_clean":"2024-09-15T21:14:29.704439+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:29.704439+0000","last_undegraded":"2024-09-15T21:14:29.704439+0000","last_fullsized":"2024-09-15T21:14:29.704439+0000","mapping_epoch":36,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4276224,"num_objects":522,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":240,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11915","reported_seq":11983,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:30.717192+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:30.717192+0000","last_peered":"2024-09-15T21:14:30.717192+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:30.717192+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4232460,"num_objects":517,"num_object_clones":0,"num_object_copies":2068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11913,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1762}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12100","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.430821+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:31.430821+0000","last_peered":"2024-09-15T21:14:31.430821+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:31.430821+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756748,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12024","reported_seq":12563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.544247+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:33.544247+0000","last_peered":"2024-09-15T21:14:33.544247+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:33.544247+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297996,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67848092,"num_objects":8287,"num_object_clones":0,"num_object_copies":33148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8287,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165053,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":135970816,"data_stored":135970816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160935,"ondisk_log_size":160935,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":719828,"kb_used_data":34244,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93652012,"statfs":{"total":96636764160,"available":95899660288,"internally_reserved":0,"allocated":35065856,"data_stored":34884567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,4,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727340,"kb_used_data":33948,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93644500,"statfs":{"total":96636764160,"available":95891968000,"internally_reserved":0,"allocated":34762752,"data_stored":34581463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051710,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":580248,"kb_used_data":34952,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93791592,"statfs":{"total":96636764160,"available":96042590208,"internally_reserved":0,"allocated":35790848,"data_stored":35609559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727196,"kb_used_data":31948,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93644644,"statfs":{"total":96636764160,"available":95892115456,"internally_reserved":0,"allocated":32714752,"data_stored":32533463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32243712,"data_stored":32243712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":35348480,"data_stored":35348480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33935360,"data_stored":33935360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":34443264,"data_stored":34443264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:35.414 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11842", 'reported_seq': 11895, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.774538+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:30.774538+0000', 'last_peered': '2024-09-15T21:14:30.774538+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:30.774538+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10042, 'log_dups_size': 0, 'ondisk_log_size': 10042, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4177920, 'num_objects': 510, 'num_object_clones': 0, 'num_object_copies': 2040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 510, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11842, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1766}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11871", 'reported_seq': 11962, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:31.333026+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:31.333026+0000', 'last_peered': '2024-09-15T21:14:31.333026+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:31.333026+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10071, 'log_dups_size': 0, 'ondisk_log_size': 10071, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4594334, 'num_objects': 562, 'num_object_clones': 0, 'num_object_copies': 2248, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 562, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11870, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1696}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'11966", 'reported_seq': 12046, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:33.259598+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:33.259598+0000', 'last_peered': '2024-09-15T21:14:33.259598+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:33.259598+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4011276, 'num_objects': 490, 'num_object_clones': 0, 'num_object_copies': 1960, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11964, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1844}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12165", 'reported_seq': 12232, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.739809+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:30.739809+0000', 'last_peered': '2024-09-15T21:14:30.739809+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:30.739809+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4044044, 'num_objects': 494, 'num_object_clones': 0, 'num_object_copies': 1976, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 494, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1817}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11861", 'reported_seq': 11917, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.832854+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:30.832854+0000', 'last_peered': '2024-09-15T21:14:30.832854+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:30.832854+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4218880, 'num_objects': 515, 'num_object_clones': 0, 'num_object_copies': 2060, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 515, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11861, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1801}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.415 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12286", 'reported_seq': 12364, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:33.562373+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:33.562373+0000', 'last_peered': '2024-09-15T21:14:33.562373+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:33.562373+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3978508, 'num_objects': 486, 'num_object_clones': 0, 'num_object_copies': 1944, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 486, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12284, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1880}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12262", 'reported_seq': 12342, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:33.404734+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:33.404734+0000', 'last_peered': '2024-09-15T21:14:33.404734+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:33.404734+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10062, 'log_dups_size': 0, 'ondisk_log_size': 10062, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4085004, 'num_objects': 499, 'num_object_clones': 0, 'num_object_copies': 1996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 499, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12260, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1975}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11917", 'reported_seq': 11988, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.826669+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:30.826669+0000', 'last_peered': '2024-09-15T21:14:30.826669+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:30.826669+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10017, 'log_dups_size': 0, 'ondisk_log_size': 10017, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4377112, 'num_objects': 535, 'num_object_clones': 0, 'num_object_copies': 2140, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 535, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11914, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1797}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12074", 'reported_seq': 12166, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:31.469804+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:31.469804+0000', 'last_peered': '2024-09-15T21:14:31.469804+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:31.469804+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10074, 'log_dups_size': 0, 'ondisk_log_size': 10074, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3994892, 'num_objects': 488, 'num_object_clones': 0, 'num_object_copies': 1952, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 488, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12072, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1847}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11896", 'reported_seq': 11947, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.514079+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:30.514079+0000', 'last_peered': '2024-09-15T21:14:30.514079+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:30.514079+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1800", 'ondisk_log_start': "27'1800", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4348550, 'num_objects': 531, 'num_object_clones': 0, 'num_object_copies': 2124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11895, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1725}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:35.416 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11915", 'reported_seq': 11983, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:30.717192+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:30.717192+0000', 'last_peered': '2024-09-15T21:14:30.717192+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:30.717192+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4232460, 'num_objects': 517, 'num_object_clones': 0, 'num_object_copies': 2068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 517, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11913, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1762}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:35.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:35.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12100", 'reported_seq': 12194, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:31.430821+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:31.430821+0000', 'last_peered': '2024-09-15T21:14:31.430821+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:31.430821+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4756748, 'num_objects': 581, 'num_object_clones': 0, 'num_object_copies': 2324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 581, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12098, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1700}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:35.417 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:35.417 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12024", 'reported_seq': 12563, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:33.544247+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:33.544247+0000', 'last_peered': '2024-09-15T21:14:33.544247+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:33.544247+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4297996, 'num_objects': 525, 'num_object_clones': 0, 'num_object_copies': 2100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2088, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 525, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12022, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 231, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1563}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:35.417 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:36.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:36.160 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29925/33148 objects degraded (90.277%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":741,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8287,"data_bytes":67848092,"bytes_used":2820722688,"bytes_avail":383726333952,"bytes_total":386547056640,"degraded_objects":29925,"degraded_total":33148,"degraded_ratio":0.90276939785205745,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:36.215 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:36.215 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:36.789 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:36.790 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:36.838 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-15T21:14:36.195853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":63399836,"num_objects":7744,"num_object_clones":0,"num_object_copies":30976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165596,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2738488,"kb_used_data":127288,"kb_used_omap":14,"kb_used_meta":2611121,"kb_avail":374748872,"statfs":{"total":386547056640,"available":383742844928,"internally_reserved":0,"allocated":130342912,"data_stored":129617756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2673788466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,12,8,14,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20127744,"num_objects":-2457,"num_object_clones":0,"num_object_copies":-9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-76,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2457,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004031"},"pg_stats":[{"pgid":"1.f","version":"37'11906","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.851284+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:35.851284+0000","last_peered":"2024-09-15T21:14:35.851284+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:35.851284+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12063","reported_seq":16961,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:34.699669+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:34.699669+0000","last_peered":"2024-09-15T21:14:34.699669+0000","last_clean":"2024-09-15T21:14:34.699669+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:34.699669+0000","last_undegraded":"2024-09-15T21:14:34.699669+0000","last_fullsized":"2024-09-15T21:14:34.699669+0000","mapping_epoch":34,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3989504,"num_objects":487,"num_object_clones":0,"num_object_copies":1948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11871","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.333026+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:31.333026+0000","last_peered":"2024-09-15T21:14:31.333026+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:31.333026+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1800","ondisk_log_start":"27'1800","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4594334,"num_objects":562,"num_object_clones":0,"num_object_copies":2248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11870,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1696}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'11966","reported_seq":12046,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.259598+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:33.259598+0000","last_peered":"2024-09-15T21:14:33.259598+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:33.259598+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4011276,"num_objects":490,"num_object_clones":0,"num_object_copies":1960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1844}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12233","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.790433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:35.790433+0000","last_peered":"2024-09-15T21:14:35.790433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:35.790433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11915","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.534399+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:35.534399+0000","last_peered":"2024-09-15T21:14:35.534399+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:35.534399+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3776512,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12286","reported_seq":12364,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.562373+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:33.562373+0000","last_peered":"2024-09-15T21:14:33.562373+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:33.562373+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3978508,"num_objects":486,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12284,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1880}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12262","reported_seq":12342,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.404734+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:33.404734+0000","last_peered":"2024-09-15T21:14:33.404734+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:33.404734+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4085004,"num_objects":499,"num_object_clones":0,"num_object_copies":1996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12260,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1975}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11986","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.772035+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:35.772035+0000","last_peered":"2024-09-15T21:14:35.772035+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:35.772035+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3811864,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12074","reported_seq":12166,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.469804+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:31.469804+0000","last_peered":"2024-09-15T21:14:31.469804+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:31.469804+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3994892,"num_objects":488,"num_object_clones":0,"num_object_copies":1952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12072,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1847}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":16591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:35.879028+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:35.879028+0000","last_peered":"2024-09-15T21:14:35.879028+0000","last_clean":"2024-09-15T21:14:35.879028+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:35.879028+0000","last_undegraded":"2024-09-15T21:14:35.879028+0000","last_fullsized":"2024-09-15T21:14:35.879028+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3629056,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11969","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.679993+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:35.679993+0000","last_peered":"2024-09-15T21:14:35.679993+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:35.679993+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3750534,"num_objects":458,"num_object_clones":0,"num_object_copies":1832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12034","reported_seq":16728,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:34.714961+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:34.714961+0000","last_peered":"2024-09-15T21:14:34.714961+0000","last_clean":"2024-09-15T21:14:34.714961+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:34.714961+0000","last_undegraded":"2024-09-15T21:14:34.714961+0000","last_fullsized":"2024-09-15T21:14:34.714961+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3817472,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11972","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.822027+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:35.822027+0000","last_peered":"2024-09-15T21:14:35.822027+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:35.822027+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12100","reported_seq":12194,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:31.430821+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:31.430821+0000","last_peered":"2024-09-15T21:14:31.430821+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:31.430821+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4756748,"num_objects":581,"num_object_clones":0,"num_object_copies":2324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12024","reported_seq":12563,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:33.544247+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:33.544247+0000","last_peered":"2024-09-15T21:14:33.544247+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:33.544247+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4297996,"num_objects":525,"num_object_clones":0,"num_object_copies":2100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2088,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1563}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":63399836,"num_objects":7744,"num_object_clones":0,"num_object_copies":30976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29925,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165596,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":127963136,"data_stored":127963136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160878,"ondisk_log_size":160878,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720076,"kb_used_data":30332,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93651764,"statfs":{"total":96636764160,"available":95899406336,"internally_reserved":0,"allocated":31059968,"data_stored":30878679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727340,"kb_used_data":33948,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":93644500,"statfs":{"total":96636764160,"available":95891968000,"internally_reserved":0,"allocated":34762752,"data_stored":34581463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":563876,"kb_used_data":31060,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93807964,"statfs":{"total":96636764160,"available":96059355136,"internally_reserved":0,"allocated":31805440,"data_stored":31624151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,4,3,7,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247949,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727196,"kb_used_data":31948,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93644644,"statfs":{"total":96636764160,"available":95892115456,"internally_reserved":0,"allocated":32714752,"data_stored":32533463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,2,2,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32243712,"data_stored":32243712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":31342592,"data_stored":31342592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":33935360,"data_stored":33935360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30441472,"data_stored":30441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:39.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:40.427 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:40.429 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:40.455 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-15T21:14:40.197465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":58927004,"num_objects":7198,"num_object_clones":0,"num_object_copies":28792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166142,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722768,"kb_used_data":115728,"kb_used_omap":14,"kb_used_meta":2606961,"kb_avail":374764592,"statfs":{"total":386547056640,"available":383758942208,"internally_reserved":0,"allocated":118505472,"data_stored":117780316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2669528626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,10,10,9,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18743296,"num_objects":-2288,"num_object_clones":0,"num_object_copies":-9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2288,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004296"},"pg_stats":[{"pgid":"1.f","version":"37'11906","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.851284+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:35.851284+0000","last_peered":"2024-09-15T21:14:35.851284+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:35.851284+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12127","reported_seq":17025,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.818043+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:39.818043+0000","last_peered":"2024-09-15T21:14:39.818043+0000","last_clean":"2024-09-15T21:14:39.818043+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:39.818043+0000","last_undegraded":"2024-09-15T21:14:39.818043+0000","last_fullsized":"2024-09-15T21:14:39.818043+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3465216,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11932","reported_seq":12023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.406564+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:36.406564+0000","last_peered":"2024-09-15T21:14:36.406564+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:36.406564+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4094622,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11931,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.552398+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:38.552398+0000","last_peered":"2024-09-15T21:14:38.552398+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:38.552398+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618060,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12233","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.790433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:35.790433+0000","last_peered":"2024-09-15T21:14:35.790433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:35.790433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11915","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.534399+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:35.534399+0000","last_peered":"2024-09-15T21:14:35.534399+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:35.534399+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3776512,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12352","reported_seq":12433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.479239+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:38.479239+0000","last_peered":"2024-09-15T21:14:38.479239+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:38.479239+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437836,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12311","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.165615+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:38.165615+0000","last_peered":"2024-09-15T21:14:38.165615+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:38.165615+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683596,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11986","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.772035+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:35.772035+0000","last_peered":"2024-09-15T21:14:35.772035+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:35.772035+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3811864,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12140","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.398954+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:36.398954+0000","last_peered":"2024-09-15T21:14:36.398954+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:36.398954+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3454220,"num_objects":422,"num_object_clones":0,"num_object_copies":1688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":16591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:35.879028+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:35.879028+0000","last_peered":"2024-09-15T21:14:35.879028+0000","last_clean":"2024-09-15T21:14:35.879028+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:35.879028+0000","last_undegraded":"2024-09-15T21:14:35.879028+0000","last_fullsized":"2024-09-15T21:14:35.879028+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3629056,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11969","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.679993+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:35.679993+0000","last_peered":"2024-09-15T21:14:35.679993+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:35.679993+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3750534,"num_objects":458,"num_object_clones":0,"num_object_copies":1832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12096","reported_seq":16788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.707058+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:39.707058+0000","last_peered":"2024-09-15T21:14:39.707058+0000","last_clean":"2024-09-15T21:14:39.707058+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:39.707058+0000","last_undegraded":"2024-09-15T21:14:39.707058+0000","last_fullsized":"2024-09-15T21:14:39.707058+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11972","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.822027+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:35.822027+0000","last_peered":"2024-09-15T21:14:35.822027+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:35.822027+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12165","reported_seq":12259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.206050+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:36.206050+0000","last_peered":"2024-09-15T21:14:36.206050+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:36.206050+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4224268,"num_objects":516,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1765}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12089","reported_seq":12704,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.531233+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:38.531233+0000","last_peered":"2024-09-15T21:14:38.531233+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:38.531233+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":58927004,"num_objects":7198,"num_object_clones":0,"num_object_copies":28792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166142,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115916800,"data_stored":115916800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720076,"kb_used_data":30332,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93651764,"statfs":{"total":96636764160,"available":95899406336,"internally_reserved":0,"allocated":31059968,"data_stored":30878679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710956,"kb_used_data":30044,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93660884,"statfs":{"total":96636764160,"available":95908745216,"internally_reserved":0,"allocated":30765056,"data_stored":30583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,0,2,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564176,"kb_used_data":27200,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93807664,"statfs":{"total":96636764160,"available":96059047936,"internally_reserved":0,"allocated":27852800,"data_stored":27671511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727560,"kb_used_data":28152,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93644280,"statfs":{"total":96636764160,"available":95891742720,"internally_reserved":0,"allocated":28827648,"data_stored":28646359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28282880,"data_stored":28282880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27246592,"data_stored":27246592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29945856,"data_stored":29945856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30441472,"data_stored":30441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:40.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:41.043 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:41.044 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:41.072 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-15T21:14:40.197465+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":58927004,"num_objects":7198,"num_object_clones":0,"num_object_copies":28792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166142,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2722768,"kb_used_data":115728,"kb_used_omap":14,"kb_used_meta":2606961,"kb_avail":374764592,"statfs":{"total":386547056640,"available":383758942208,"internally_reserved":0,"allocated":118505472,"data_stored":117780316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2669528626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,4,6,10,10,9,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18743296,"num_objects":-2288,"num_object_clones":0,"num_object_copies":-9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2288,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004296"},"pg_stats":[{"pgid":"1.f","version":"37'11906","reported_seq":11962,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.851284+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:35.851284+0000","last_peered":"2024-09-15T21:14:35.851284+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:35.851284+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3653632,"num_objects":446,"num_object_clones":0,"num_object_copies":1784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11906,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1830}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12127","reported_seq":17025,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.818043+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:39.818043+0000","last_peered":"2024-09-15T21:14:39.818043+0000","last_clean":"2024-09-15T21:14:39.818043+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:39.818043+0000","last_undegraded":"2024-09-15T21:14:39.818043+0000","last_fullsized":"2024-09-15T21:14:39.818043+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3465216,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11932","reported_seq":12023,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.406564+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:36.406564+0000","last_peered":"2024-09-15T21:14:36.406564+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:36.406564+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4094622,"num_objects":501,"num_object_clones":0,"num_object_copies":2004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11931,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1757}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.552398+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:38.552398+0000","last_peered":"2024-09-15T21:14:38.552398+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:38.552398+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618060,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12233","reported_seq":12299,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.790433+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:35.790433+0000","last_peered":"2024-09-15T21:14:35.790433+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:35.790433+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3486988,"num_objects":426,"num_object_clones":0,"num_object_copies":1704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12231,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11915","reported_seq":11971,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.534399+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:35.534399+0000","last_peered":"2024-09-15T21:14:35.534399+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:35.534399+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3776512,"num_objects":461,"num_object_clones":0,"num_object_copies":1844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1855}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12352","reported_seq":12433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.479239+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:38.479239+0000","last_peered":"2024-09-15T21:14:38.479239+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:38.479239+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437836,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12311","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.165615+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:38.165615+0000","last_peered":"2024-09-15T21:14:38.165615+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:38.165615+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683596,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'11986","reported_seq":12054,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.772035+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:35.772035+0000","last_peered":"2024-09-15T21:14:35.772035+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:35.772035+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3811864,"num_objects":466,"num_object_clones":0,"num_object_copies":1864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11983,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1866}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12140","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.398954+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:36.398954+0000","last_peered":"2024-09-15T21:14:36.398954+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:36.398954+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3454220,"num_objects":422,"num_object_clones":0,"num_object_copies":1688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12138,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1913}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11917","reported_seq":16591,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:35.879028+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:35.879028+0000","last_peered":"2024-09-15T21:14:35.879028+0000","last_clean":"2024-09-15T21:14:35.879028+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:35.879028+0000","last_undegraded":"2024-09-15T21:14:35.879028+0000","last_fullsized":"2024-09-15T21:14:35.879028+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3629056,"num_objects":443,"num_object_clones":0,"num_object_copies":1772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'11969","reported_seq":12020,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.679993+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:35.679993+0000","last_peered":"2024-09-15T21:14:35.679993+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:35.679993+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3750534,"num_objects":458,"num_object_clones":0,"num_object_copies":1832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1798}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12096","reported_seq":16788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.707058+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:39.707058+0000","last_peered":"2024-09-15T21:14:39.707058+0000","last_clean":"2024-09-15T21:14:39.707058+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:39.707058+0000","last_undegraded":"2024-09-15T21:14:39.707058+0000","last_fullsized":"2024-09-15T21:14:39.707058+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'11972","reported_seq":12040,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:35.822027+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:35.822027+0000","last_peered":"2024-09-15T21:14:35.822027+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:35.822027+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11970,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1819}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12165","reported_seq":12259,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:36.206050+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:36.206050+0000","last_peered":"2024-09-15T21:14:36.206050+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:36.206050+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4224268,"num_objects":516,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12163,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1765}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12089","reported_seq":12704,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.531233+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:38.531233+0000","last_peered":"2024-09-15T21:14:38.531233+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:38.531233+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":58927004,"num_objects":7198,"num_object_clones":0,"num_object_copies":28792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166142,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115916800,"data_stored":115916800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160724,"ondisk_log_size":160724,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720076,"kb_used_data":30332,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93651764,"statfs":{"total":96636764160,"available":95899406336,"internally_reserved":0,"allocated":31059968,"data_stored":30878679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,3,5,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":710956,"kb_used_data":30044,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93660884,"statfs":{"total":96636764160,"available":95908745216,"internally_reserved":0,"allocated":30765056,"data_stored":30583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,0,2,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564176,"kb_used_data":27200,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93807664,"statfs":{"total":96636764160,"available":96059047936,"internally_reserved":0,"allocated":27852800,"data_stored":27671511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727560,"kb_used_data":28152,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93644280,"statfs":{"total":96636764160,"available":95891742720,"internally_reserved":0,"allocated":28827648,"data_stored":28646359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28282880,"data_stored":28282880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27246592,"data_stored":27246592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":29945856,"data_stored":29945856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":30441472,"data_stored":30441472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'11906", 'reported_seq': 11962, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.851284+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:35.851284+0000', 'last_peered': '2024-09-15T21:14:35.851284+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:35.851284+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10006, 'log_dups_size': 0, 'ondisk_log_size': 10006, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3653632, 'num_objects': 446, 'num_object_clones': 0, 'num_object_copies': 1784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 446, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11906, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1830}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11932", 'reported_seq': 12023, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:36.406564+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:36.406564+0000', 'last_peered': '2024-09-15T21:14:36.406564+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:36.406564+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4094622, 'num_objects': 501, 'num_object_clones': 0, 'num_object_copies': 2004, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 501, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11931, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1757}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12014", 'reported_seq': 12094, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:38.552398+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:38.552398+0000', 'last_peered': '2024-09-15T21:14:38.552398+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:38.552398+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10014, 'log_dups_size': 0, 'ondisk_log_size': 10014, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3618060, 'num_objects': 442, 'num_object_clones': 0, 'num_object_copies': 1768, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 442, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12012, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1892}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:41.073 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12233", 'reported_seq': 12299, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.790433+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:35.790433+0000', 'last_peered': '2024-09-15T21:14:35.790433+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:35.790433+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10033, 'log_dups_size': 0, 'ondisk_log_size': 10033, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3486988, 'num_objects': 426, 'num_object_clones': 0, 'num_object_copies': 1704, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 426, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12231, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1885}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'11915", 'reported_seq': 11971, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.534399+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:35.534399+0000', 'last_peered': '2024-09-15T21:14:35.534399+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:35.534399+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3776512, 'num_objects': 461, 'num_object_clones': 0, 'num_object_copies': 1844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 461, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11915, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1855}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12352", 'reported_seq': 12433, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:38.479239+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:38.479239+0000', 'last_peered': '2024-09-15T21:14:38.479239+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:38.479239+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "27'2300", 'ondisk_log_start': "27'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3437836, 'num_objects': 420, 'num_object_clones': 0, 'num_object_copies': 1680, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 420, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12350, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1946}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12311", 'reported_seq': 12391, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:38.165615+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:38.165615+0000', 'last_peered': '2024-09-15T21:14:38.165615+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:38.165615+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2300", 'ondisk_log_start': "27'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3683596, 'num_objects': 450, 'num_object_clones': 0, 'num_object_copies': 1800, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 450, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12309, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2024}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'11986", 'reported_seq': 12054, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.772035+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:35.772035+0000', 'last_peered': '2024-09-15T21:14:35.772035+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:35.772035+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10086, 'log_dups_size': 0, 'ondisk_log_size': 10086, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3811864, 'num_objects': 466, 'num_object_clones': 0, 'num_object_copies': 1864, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 466, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11983, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1866}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:41.074 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12140", 'reported_seq': 12232, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:36.398954+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:36.398954+0000', 'last_peered': '2024-09-15T21:14:36.398954+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:36.398954+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10040, 'log_dups_size': 0, 'ondisk_log_size': 10040, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3454220, 'num_objects': 422, 'num_object_clones': 0, 'num_object_copies': 1688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 422, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12138, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1913}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'11969", 'reported_seq': 12020, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.679993+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:35.679993+0000', 'last_peered': '2024-09-15T21:14:35.679993+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:35.679993+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10069, 'log_dups_size': 0, 'ondisk_log_size': 10069, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3750534, 'num_objects': 458, 'num_object_clones': 0, 'num_object_copies': 1832, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 458, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11968, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1798}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'11972", 'reported_seq': 12040, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:35.822027+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:35.822027+0000', 'last_peered': '2024-09-15T21:14:35.822027+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:35.822027+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10072, 'log_dups_size': 0, 'ondisk_log_size': 10072, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3765516, 'num_objects': 460, 'num_object_clones': 0, 'num_object_copies': 1840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 11970, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1819}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12165", 'reported_seq': 12259, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:36.206050+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:36.206050+0000', 'last_peered': '2024-09-15T21:14:36.206050+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:36.206050+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4224268, 'num_objects': 516, 'num_object_clones': 0, 'num_object_copies': 2064, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 516, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12163, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1765}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:41.075 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12089", 'reported_seq': 12704, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:38.531233+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:38.531233+0000', 'last_peered': '2024-09-15T21:14:38.531233+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:38.531233+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3765516, 'num_objects': 460, 'num_object_clones': 0, 'num_object_copies': 1840, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2050, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 460, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12087, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 269, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1590}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:41.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:41.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:41.800 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29887/28792 objects degraded (103.803%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":747,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7198,"data_bytes":58927004,"bytes_used":2788114432,"bytes_avail":383758942208,"bytes_total":386547056640,"degraded_objects":29887,"degraded_total":28792,"degraded_ratio":1.0380313976104474,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":190},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:41.833 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:41.833 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:42.513 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:42.515 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:42.536 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2024-09-15T21:14:42.198328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":53864348,"num_objects":6580,"num_object_clones":0,"num_object_copies":26320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166760,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2706764,"kb_used_data":108024,"kb_used_omap":14,"kb_used_meta":2598641,"kb_avail":374780596,"statfs":{"total":386547056640,"available":383775330304,"internally_reserved":0,"allocated":110616576,"data_stored":109891420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2661008946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,14,9,7,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20938752,"num_objects":-2556,"num_object_clones":0,"num_object_copies":-10224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-72,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2556,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004517"},"pg_stats":[{"pgid":"1.f","version":"37'11964","reported_seq":12021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.856223+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:40.856223+0000","last_peered":"2024-09-15T21:14:40.856223+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:40.856223+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3178496,"num_objects":388,"num_object_clones":0,"num_object_copies":1552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12127","reported_seq":17025,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.818043+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:39.818043+0000","last_peered":"2024-09-15T21:14:39.818043+0000","last_clean":"2024-09-15T21:14:39.818043+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:39.818043+0000","last_undegraded":"2024-09-15T21:14:39.818043+0000","last_fullsized":"2024-09-15T21:14:39.818043+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3465216,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11999","reported_seq":12090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.557714+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:41.557714+0000","last_peered":"2024-09-15T21:14:41.557714+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:41.557714+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3545758,"num_objects":434,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11998,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12014","reported_seq":12094,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.552398+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:38.552398+0000","last_peered":"2024-09-15T21:14:38.552398+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:38.552398+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3618060,"num_objects":442,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1892}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12288","reported_seq":12355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.634680+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:40.634680+0000","last_peered":"2024-09-15T21:14:40.634680+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:40.634680+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3036428,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11985","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.839616+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:40.839616+0000","last_peered":"2024-09-15T21:14:40.839616+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:40.839616+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3203072,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12352","reported_seq":12433,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.479239+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:38.479239+0000","last_peered":"2024-09-15T21:14:38.479239+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:38.479239+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3437836,"num_objects":420,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12350,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1946}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12311","reported_seq":12391,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.165615+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:38.165615+0000","last_peered":"2024-09-15T21:14:38.165615+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:38.165615+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3683596,"num_objects":450,"num_object_clones":0,"num_object_copies":1800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12309,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2024}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12047","reported_seq":12118,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.769197+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:40.769197+0000","last_peered":"2024-09-15T21:14:40.769197+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:40.769197+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3312152,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12195","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.371149+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:41.371149+0000","last_peered":"2024-09-15T21:14:41.371149+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:41.371149+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3003660,"num_objects":367,"num_object_clones":0,"num_object_copies":1468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11970","reported_seq":16644,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:40.853455+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:40.853455+0000","last_peered":"2024-09-15T21:14:40.853455+0000","last_clean":"2024-09-15T21:14:40.853455+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:40.853455+0000","last_undegraded":"2024-09-15T21:14:40.853455+0000","last_fullsized":"2024-09-15T21:14:40.853455+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3194880,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6081,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12018","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.929500+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:40.929500+0000","last_peered":"2024-09-15T21:14:40.929500+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:40.929500+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3349126,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12096","reported_seq":16788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.707058+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:39.707058+0000","last_peered":"2024-09-15T21:14:39.707058+0000","last_clean":"2024-09-15T21:14:39.707058+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:39.707058+0000","last_undegraded":"2024-09-15T21:14:39.707058+0000","last_fullsized":"2024-09-15T21:14:39.707058+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12034","reported_seq":12102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.991417+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:40.991417+0000","last_peered":"2024-09-15T21:14:40.991417+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:40.991417+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3257612,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12253","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.579381+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:41.579381+0000","last_peered":"2024-09-15T21:14:41.579381+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:41.579381+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3503372,"num_objects":428,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12089","reported_seq":12704,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:38.531233+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:38.531233+0000","last_peered":"2024-09-15T21:14:38.531233+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:38.531233+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3765516,"num_objects":460,"num_object_clones":0,"num_object_copies":1840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12087,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":269,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1590}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":53864348,"num_objects":6580,"num_object_clones":0,"num_object_copies":26320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29887,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":166760,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107831296,"data_stored":107831296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160942,"ondisk_log_size":160942,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703760,"kb_used_data":26484,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93668080,"statfs":{"total":96636764160,"available":95916113920,"internally_reserved":0,"allocated":27119616,"data_stored":26938327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,0,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711268,"kb_used_data":26188,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93660572,"statfs":{"total":96636764160,"available":95908425728,"internally_reserved":0,"allocated":26816512,"data_stored":26635223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564176,"kb_used_data":27200,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93807664,"statfs":{"total":96636764160,"available":96059047936,"internally_reserved":0,"allocated":27852800,"data_stored":27671511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247950,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727560,"kb_used_data":28152,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93644280,"statfs":{"total":96636764160,"available":95891742720,"internally_reserved":0,"allocated":28827648,"data_stored":28646359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,3,3,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28282880,"data_stored":28282880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27246592,"data_stored":27246592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25944064,"data_stored":25944064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26357760,"data_stored":26357760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:45.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:46.131 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:46.132 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:46.158 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-15T21:14:44.199206+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":51693468,"num_objects":6315,"num_object_clones":0,"num_object_copies":25260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167025,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707012,"kb_used_data":104112,"kb_used_omap":14,"kb_used_meta":2602801,"kb_avail":374780348,"statfs":{"total":386547056640,"available":383775076352,"internally_reserved":0,"allocated":106610688,"data_stored":105885532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2665268786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,6,19,14,5,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18112512,"num_objects":-2211,"num_object_clones":0,"num_object_copies":-8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-121,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2211,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004812"},"pg_stats":[{"pgid":"1.f","version":"37'11964","reported_seq":12021,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.856223+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:40.856223+0000","last_peered":"2024-09-15T21:14:40.856223+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:40.856223+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3178496,"num_objects":388,"num_object_clones":0,"num_object_copies":1552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1888}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12127","reported_seq":17025,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.818043+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:39.818043+0000","last_peered":"2024-09-15T21:14:39.818043+0000","last_clean":"2024-09-15T21:14:39.818043+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:39.818043+0000","last_undegraded":"2024-09-15T21:14:39.818043+0000","last_fullsized":"2024-09-15T21:14:39.818043+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3465216,"num_objects":423,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2793,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11999","reported_seq":12090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.557714+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:41.557714+0000","last_peered":"2024-09-15T21:14:41.557714+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:41.557714+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3545758,"num_objects":434,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11998,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12087","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.328081+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:43.328081+0000","last_peered":"2024-09-15T21:14:43.328081+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:43.328081+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3020044,"num_objects":369,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12085,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12288","reported_seq":12355,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.634680+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:40.634680+0000","last_peered":"2024-09-15T21:14:40.634680+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:40.634680+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3036428,"num_objects":371,"num_object_clones":0,"num_object_copies":1484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'11985","reported_seq":12038,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.839616+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:40.839616+0000","last_peered":"2024-09-15T21:14:40.839616+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:40.839616+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3203072,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1925}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12424","reported_seq":12504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.697885+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:43.697885+0000","last_peered":"2024-09-15T21:14:43.697885+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:43.697885+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2848012,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12422,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12370","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.372040+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:43.372040+0000","last_peered":"2024-09-15T21:14:43.372040+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:43.372040+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3200268,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12368,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12047","reported_seq":12118,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.769197+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:40.769197+0000","last_peered":"2024-09-15T21:14:40.769197+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:40.769197+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3312152,"num_objects":405,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12044,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1927}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12195","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.371149+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:41.371149+0000","last_peered":"2024-09-15T21:14:41.371149+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:41.371149+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3003660,"num_objects":367,"num_object_clones":0,"num_object_copies":1468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'11970","reported_seq":16644,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:40.853455+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:40.853455+0000","last_peered":"2024-09-15T21:14:40.853455+0000","last_clean":"2024-09-15T21:14:40.853455+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:40.853455+0000","last_undegraded":"2024-09-15T21:14:40.853455+0000","last_fullsized":"2024-09-15T21:14:40.853455+0000","mapping_epoch":32,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3194880,"num_objects":390,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6081,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12018","reported_seq":12069,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.929500+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:40.929500+0000","last_peered":"2024-09-15T21:14:40.929500+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:40.929500+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3349126,"num_objects":409,"num_object_clones":0,"num_object_copies":1636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12017,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1847}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12096","reported_seq":16788,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:39.707058+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:39.707058+0000","last_peered":"2024-09-15T21:14:39.707058+0000","last_clean":"2024-09-15T21:14:39.707058+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:39.707058+0000","last_undegraded":"2024-09-15T21:14:39.707058+0000","last_fullsized":"2024-09-15T21:14:39.707058+0000","mapping_epoch":36,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3309568,"num_objects":404,"num_object_clones":0,"num_object_copies":1616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12034","reported_seq":12102,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:40.991417+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:40.991417+0000","last_peered":"2024-09-15T21:14:40.991417+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:40.991417+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3257612,"num_objects":398,"num_object_clones":0,"num_object_copies":1592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1881}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12253","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.579381+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:41.579381+0000","last_peered":"2024-09-15T21:14:41.579381+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:41.579381+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3503372,"num_objects":428,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12150","reported_seq":12862,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.688545+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:43.688545+0000","last_peered":"2024-09-15T21:14:43.688545+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:43.688545+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3265804,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1602}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":51693468,"num_objects":6315,"num_object_clones":0,"num_object_copies":25260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6315,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167025,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":103825408,"data_stored":103825408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161007,"ondisk_log_size":161007,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703760,"kb_used_data":26484,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93668080,"statfs":{"total":96636764160,"available":95916113920,"internally_reserved":0,"allocated":27119616,"data_stored":26938327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,7,0,2,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711268,"kb_used_data":26188,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93660572,"statfs":{"total":96636764160,"available":95908425728,"internally_reserved":0,"allocated":26816512,"data_stored":26635223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051712,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564176,"kb_used_data":27200,"kb_used_omap":4,"kb_used_meta":536955,"kb_avail":93807664,"statfs":{"total":96636764160,"available":96059047936,"internally_reserved":0,"allocated":27852800,"data_stored":27671511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":549842639},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1,2,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727808,"kb_used_data":24240,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93644032,"statfs":{"total":96636764160,"available":95891488768,"internally_reserved":0,"allocated":24821760,"data_stored":24640471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24276992,"data_stored":24276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27246592,"data_stored":27246592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25944064,"data_stored":25944064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":26357760,"data_stored":26357760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:46.159 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:46.803 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:46.803 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:46.828 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-15T21:14:46.200095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47302556,"num_objects":5779,"num_object_clones":0,"num_object_copies":23116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167561,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161043,"ondisk_log_size":161043,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707560,"kb_used_data":96352,"kb_used_omap":14,"kb_used_meta":2611121,"kb_avail":374779800,"statfs":{"total":386547056640,"available":383774515200,"internally_reserved":0,"allocated":98664448,"data_stored":97939292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2673788466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,5,14,18,11,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20545536,"num_objects":-2508,"num_object_clones":0,"num_object_copies":-10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005016"},"pg_stats":[{"pgid":"1.f","version":"37'12022","reported_seq":12079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.067047+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:46.067047+0000","last_peered":"2024-09-15T21:14:46.067047+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:46.067047+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2703360,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12193","reported_seq":17091,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.809205+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:44.809205+0000","last_peered":"2024-09-15T21:14:44.809205+0000","last_clean":"2024-09-15T21:14:44.809205+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:44.809205+0000","last_undegraded":"2024-09-15T21:14:44.809205+0000","last_fullsized":"2024-09-15T21:14:44.809205+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924544,"num_objects":357,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11999","reported_seq":12090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.557714+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:41.557714+0000","last_peered":"2024-09-15T21:14:41.557714+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:41.557714+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3545758,"num_objects":434,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11998,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12087","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.328081+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:43.328081+0000","last_peered":"2024-09-15T21:14:43.328081+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:43.328081+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3020044,"num_objects":369,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12085,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12343","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.103404+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:46.103404+0000","last_peered":"2024-09-15T21:14:46.103404+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:46.103404+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585868,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12052","reported_seq":12108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:45.937988+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:45.937988+0000","last_peered":"2024-09-15T21:14:45.937988+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:45.937988+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12424","reported_seq":12504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.697885+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:43.697885+0000","last_peered":"2024-09-15T21:14:43.697885+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:43.697885+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2848012,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12422,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12370","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.372040+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:43.372040+0000","last_peered":"2024-09-15T21:14:43.372040+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:43.372040+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3200268,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12368,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12100","reported_seq":12171,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.055265+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:46.055265+0000","last_peered":"2024-09-15T21:14:46.055265+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:46.055265+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2877976,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12195","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.371149+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:41.371149+0000","last_peered":"2024-09-15T21:14:41.371149+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:41.371149+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3003660,"num_objects":367,"num_object_clones":0,"num_object_copies":1468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12023","reported_seq":16697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:45.889771+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:45.889771+0000","last_peered":"2024-09-15T21:14:45.889771+0000","last_clean":"2024-09-15T21:14:45.889771+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:45.889771+0000","last_undegraded":"2024-09-15T21:14:45.889771+0000","last_fullsized":"2024-09-15T21:14:45.889771+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12081","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.058995+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:46.058995+0000","last_peered":"2024-09-15T21:14:46.058995+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:46.058995+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2833030,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12080,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12154","reported_seq":16848,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.950317+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:44.950317+0000","last_peered":"2024-09-15T21:14:44.950317+0000","last_clean":"2024-09-15T21:14:44.950317+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:44.950317+0000","last_undegraded":"2024-09-15T21:14:44.950317+0000","last_fullsized":"2024-09-15T21:14:44.950317+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12097","reported_seq":12165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.004045+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:46.004045+0000","last_peered":"2024-09-15T21:14:46.004045+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:46.004045+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741516,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12253","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.579381+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:41.579381+0000","last_peered":"2024-09-15T21:14:41.579381+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:41.579381+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3503372,"num_objects":428,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12150","reported_seq":12862,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.688545+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:43.688545+0000","last_peered":"2024-09-15T21:14:43.688545+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:43.688545+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3265804,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1602}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47302556,"num_objects":5779,"num_object_clones":0,"num_object_copies":23116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167561,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95731712,"data_stored":95731712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161043,"ondisk_log_size":161043,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703956,"kb_used_data":22532,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93667884,"statfs":{"total":96636764160,"available":95915913216,"internally_reserved":0,"allocated":23072768,"data_stored":22891479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711268,"kb_used_data":26188,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93660572,"statfs":{"total":96636764160,"available":95908425728,"internally_reserved":0,"allocated":26816512,"data_stored":26635223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564528,"kb_used_data":23392,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93807312,"statfs":{"total":96636764160,"available":96058687488,"internally_reserved":0,"allocated":23953408,"data_stored":23772119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,3,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727808,"kb_used_data":24240,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93644032,"statfs":{"total":96636764160,"available":95891488768,"internally_reserved":0,"allocated":24821760,"data_stored":24640471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24276992,"data_stored":24276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23232512,"data_stored":23232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25944064,"data_stored":25944064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22278144,"data_stored":22278144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:46.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:46.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12022", 'reported_seq': 12079, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:46.067047+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:46.067047+0000', 'last_peered': '2024-09-15T21:14:46.067047+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:46.067047+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10022, 'log_dups_size': 0, 'ondisk_log_size': 10022, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2703360, 'num_objects': 330, 'num_object_clones': 0, 'num_object_copies': 1320, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 330, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12022, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1946}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:46.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'11999", 'reported_seq': 12090, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:41.557714+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:41.557714+0000', 'last_peered': '2024-09-15T21:14:41.557714+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:41.557714+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'1900", 'ondisk_log_start': "27'1900", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3545758, 'num_objects': 434, 'num_object_clones': 0, 'num_object_copies': 1736, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 434, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 11998, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1824}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:46.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12087", 'reported_seq': 12167, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:43.328081+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:43.328081+0000', 'last_peered': '2024-09-15T21:14:43.328081+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:43.328081+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3020044, 'num_objects': 369, 'num_object_clones': 0, 'num_object_copies': 1476, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 369, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12085, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1965}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12343", 'reported_seq': 12410, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:46.103404+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:46.103404+0000', 'last_peered': '2024-09-15T21:14:46.103404+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:46.103404+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2585868, 'num_objects': 316, 'num_object_clones': 0, 'num_object_copies': 1264, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 316, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12341, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1995}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12052", 'reported_seq': 12108, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:45.937988+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:45.937988+0000', 'last_peered': '2024-09-15T21:14:45.937988+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:45.937988+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2654208, 'num_objects': 324, 'num_object_clones': 0, 'num_object_copies': 1296, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 324, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12052, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1992}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12424", 'reported_seq': 12504, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:43.697885+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:43.697885+0000', 'last_peered': '2024-09-15T21:14:43.697885+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:43.697885+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10024, 'log_dups_size': 0, 'ondisk_log_size': 10024, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2848012, 'num_objects': 348, 'num_object_clones': 0, 'num_object_copies': 1392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 348, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12422, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2018}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12370", 'reported_seq': 12450, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:43.372040+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:43.372040+0000', 'last_peered': '2024-09-15T21:14:43.372040+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:43.372040+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2300", 'ondisk_log_start': "27'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3200268, 'num_objects': 391, 'num_object_clones': 0, 'num_object_copies': 1564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 391, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12368, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2083}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:46.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12100", 'reported_seq': 12171, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:46.055265+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:46.055265+0000', 'last_peered': '2024-09-15T21:14:46.055265+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:46.055265+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2877976, 'num_objects': 352, 'num_object_clones': 0, 'num_object_copies': 1408, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 352, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12097, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1980}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12195", 'reported_seq': 12287, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:41.371149+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:41.371149+0000', 'last_peered': '2024-09-15T21:14:41.371149+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:41.371149+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10095, 'log_dups_size': 0, 'ondisk_log_size': 10095, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3003660, 'num_objects': 367, 'num_object_clones': 0, 'num_object_copies': 1468, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 367, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12193, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1968}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12081", 'reported_seq': 12131, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:46.058995+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:46.058995+0000', 'last_peered': '2024-09-15T21:14:46.058995+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:46.058995+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2833030, 'num_objects': 346, 'num_object_clones': 0, 'num_object_copies': 1384, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 346, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12080, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1910}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12097", 'reported_seq': 12165, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:46.004045+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:46.004045+0000', 'last_peered': '2024-09-15T21:14:46.004045+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:46.004045+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10097, 'log_dups_size': 0, 'ondisk_log_size': 10097, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2741516, 'num_objects': 335, 'num_object_clones': 0, 'num_object_copies': 1340, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 335, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12095, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1944}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12253", 'reported_seq': 12347, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:41.579381+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:41.579381+0000', 'last_peered': '2024-09-15T21:14:41.579381+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:41.579381+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10053, 'log_dups_size': 0, 'ondisk_log_size': 10053, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3503372, 'num_objects': 428, 'num_object_clones': 0, 'num_object_copies': 1712, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 428, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12251, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1853}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:46.831 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12150", 'reported_seq': 12862, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:43.688545+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:43.688545+0000', 'last_peered': '2024-09-15T21:14:43.688545+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:43.688545+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10050, 'log_dups_size': 0, 'ondisk_log_size': 10050, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3265804, 'num_objects': 399, 'num_object_clones': 0, 'num_object_copies': 1596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2001, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 399, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12148, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 318, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1602}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:46.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:47.594 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:47.594 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29838/23116 objects degraded (129.079%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":5779,"data_bytes":47302556,"bytes_used":2772541440,"bytes_avail":383774515200,"bytes_total":386547056640,"degraded_objects":29838,"degraded_total":23116,"degraded_ratio":1.2907942550614293,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":208},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:47.623 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:47.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:48.193 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:48.194 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:48.225 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":404,"stamp":"2024-09-15T21:14:46.200095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47302556,"num_objects":5779,"num_object_clones":0,"num_object_copies":23116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167561,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161043,"ondisk_log_size":161043,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2707560,"kb_used_data":96352,"kb_used_omap":14,"kb_used_meta":2611121,"kb_avail":374779800,"statfs":{"total":386547056640,"available":383774515200,"internally_reserved":0,"allocated":98664448,"data_stored":97939292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2673788466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,2,5,14,18,11,7,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20545536,"num_objects":-2508,"num_object_clones":0,"num_object_copies":-10032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005016"},"pg_stats":[{"pgid":"1.f","version":"37'12022","reported_seq":12079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.067047+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:46.067047+0000","last_peered":"2024-09-15T21:14:46.067047+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:46.067047+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2703360,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12193","reported_seq":17091,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.809205+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:44.809205+0000","last_peered":"2024-09-15T21:14:44.809205+0000","last_clean":"2024-09-15T21:14:44.809205+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:44.809205+0000","last_undegraded":"2024-09-15T21:14:44.809205+0000","last_fullsized":"2024-09-15T21:14:44.809205+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924544,"num_objects":357,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'11999","reported_seq":12090,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.557714+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:41.557714+0000","last_peered":"2024-09-15T21:14:41.557714+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:41.557714+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'1900","ondisk_log_start":"27'1900","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3545758,"num_objects":434,"num_object_clones":0,"num_object_copies":1736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":434,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11998,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1824}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12087","reported_seq":12167,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.328081+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:43.328081+0000","last_peered":"2024-09-15T21:14:43.328081+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:43.328081+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3020044,"num_objects":369,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12085,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1965}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12343","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.103404+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:46.103404+0000","last_peered":"2024-09-15T21:14:46.103404+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:46.103404+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585868,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12052","reported_seq":12108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:45.937988+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:45.937988+0000","last_peered":"2024-09-15T21:14:45.937988+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:45.937988+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12424","reported_seq":12504,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.697885+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:43.697885+0000","last_peered":"2024-09-15T21:14:43.697885+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:43.697885+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2848012,"num_objects":348,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12422,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12370","reported_seq":12450,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.372040+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:43.372040+0000","last_peered":"2024-09-15T21:14:43.372040+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:43.372040+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3200268,"num_objects":391,"num_object_clones":0,"num_object_copies":1564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12368,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2083}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12100","reported_seq":12171,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.055265+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:46.055265+0000","last_peered":"2024-09-15T21:14:46.055265+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:46.055265+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2877976,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12195","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.371149+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:41.371149+0000","last_peered":"2024-09-15T21:14:41.371149+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:41.371149+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3003660,"num_objects":367,"num_object_clones":0,"num_object_copies":1468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1968}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12023","reported_seq":16697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:45.889771+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:45.889771+0000","last_peered":"2024-09-15T21:14:45.889771+0000","last_clean":"2024-09-15T21:14:45.889771+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:45.889771+0000","last_undegraded":"2024-09-15T21:14:45.889771+0000","last_fullsized":"2024-09-15T21:14:45.889771+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12081","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.058995+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:46.058995+0000","last_peered":"2024-09-15T21:14:46.058995+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:46.058995+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2833030,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12080,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12154","reported_seq":16848,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.950317+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:44.950317+0000","last_peered":"2024-09-15T21:14:44.950317+0000","last_clean":"2024-09-15T21:14:44.950317+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:44.950317+0000","last_undegraded":"2024-09-15T21:14:44.950317+0000","last_fullsized":"2024-09-15T21:14:44.950317+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12097","reported_seq":12165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.004045+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:46.004045+0000","last_peered":"2024-09-15T21:14:46.004045+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:46.004045+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741516,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12253","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:41.579381+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:41.579381+0000","last_peered":"2024-09-15T21:14:41.579381+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:41.579381+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3503372,"num_objects":428,"num_object_clones":0,"num_object_copies":1712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12251,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1853}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12150","reported_seq":12862,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:43.688545+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:43.688545+0000","last_peered":"2024-09-15T21:14:43.688545+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:43.688545+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3265804,"num_objects":399,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12148,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1602}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47302556,"num_objects":5779,"num_object_clones":0,"num_object_copies":23116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5779,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":167561,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95731712,"data_stored":95731712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161043,"ondisk_log_size":161043,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703956,"kb_used_data":22532,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93667884,"statfs":{"total":96636764160,"available":95915913216,"internally_reserved":0,"allocated":23072768,"data_stored":22891479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711268,"kb_used_data":26188,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93660572,"statfs":{"total":96636764160,"available":95908425728,"internally_reserved":0,"allocated":26816512,"data_stored":26635223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564528,"kb_used_data":23392,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93807312,"statfs":{"total":96636764160,"available":96058687488,"internally_reserved":0,"allocated":23953408,"data_stored":23772119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,3,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247951,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":727808,"kb_used_data":24240,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93644032,"statfs":{"total":96636764160,"available":95891488768,"internally_reserved":0,"allocated":24821760,"data_stored":24640471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,10,8,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24276992,"data_stored":24276992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23232512,"data_stored":23232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25944064,"data_stored":25944064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22278144,"data_stored":22278144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:51.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:51.877 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:51.877 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:51.909 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":406,"stamp":"2024-09-15T21:14:50.201750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":43624348,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168010,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160892,"ondisk_log_size":160892,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704060,"kb_used_data":88680,"kb_used_omap":14,"kb_used_meta":2615281,"kb_avail":374783300,"statfs":{"total":386547056640,"available":383778099200,"internally_reserved":0,"allocated":90808320,"data_stored":90083164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2678048306},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,2,4,12,8,10,9,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18202624,"num_objects":-2222,"num_object_clones":0,"num_object_copies":-8888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-129,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005044"},"pg_stats":[{"pgid":"1.f","version":"37'12022","reported_seq":12079,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.067047+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:46.067047+0000","last_peered":"2024-09-15T21:14:46.067047+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:46.067047+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2703360,"num_objects":330,"num_object_clones":0,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12022,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12193","reported_seq":17091,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.809205+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:44.809205+0000","last_peered":"2024-09-15T21:14:44.809205+0000","last_clean":"2024-09-15T21:14:44.809205+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:44.809205+0000","last_undegraded":"2024-09-15T21:14:44.809205+0000","last_fullsized":"2024-09-15T21:14:44.809205+0000","mapping_epoch":34,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2924544,"num_objects":357,"num_object_clones":0,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2859,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12071","reported_seq":12162,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.474884+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:46.474884+0000","last_peered":"2024-09-15T21:14:46.474884+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:46.474884+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2955934,"num_objects":362,"num_object_clones":0,"num_object_copies":1448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12070,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1896}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12139","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.559045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:48.559045+0000","last_peered":"2024-09-15T21:14:48.559045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:48.559045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2594060,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12343","reported_seq":12410,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.103404+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:46.103404+0000","last_peered":"2024-09-15T21:14:46.103404+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:46.103404+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2585868,"num_objects":316,"num_object_clones":0,"num_object_copies":1264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12341,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1995}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12052","reported_seq":12108,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:45.937988+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:45.937988+0000","last_peered":"2024-09-15T21:14:45.937988+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:45.937988+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":324,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":324,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12052,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1992}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12479","reported_seq":12560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.690006+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:48.690006+0000","last_peered":"2024-09-15T21:14:48.690006+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:48.690006+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2397452,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12430","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.601178+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:48.601178+0000","last_peered":"2024-09-15T21:14:48.601178+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:48.601178+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2400","ondisk_log_start":"27'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708748,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12100","reported_seq":12171,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.055265+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:46.055265+0000","last_peered":"2024-09-15T21:14:46.055265+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:46.055265+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2877976,"num_objects":352,"num_object_clones":0,"num_object_copies":1408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1980}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12260","reported_seq":12352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.616196+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:46.616196+0000","last_peered":"2024-09-15T21:14:46.616196+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:46.616196+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2471180,"num_objects":302,"num_object_clones":0,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2033}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12023","reported_seq":16697,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:45.889771+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:45.889771+0000","last_peered":"2024-09-15T21:14:45.889771+0000","last_clean":"2024-09-15T21:14:45.889771+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:45.889771+0000","last_undegraded":"2024-09-15T21:14:45.889771+0000","last_fullsized":"2024-09-15T21:14:45.889771+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2760704,"num_objects":337,"num_object_clones":0,"num_object_copies":1348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6134,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12081","reported_seq":12131,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.058995+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:46.058995+0000","last_peered":"2024-09-15T21:14:46.058995+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:46.058995+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2833030,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12080,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1910}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12154","reported_seq":16848,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:44.950317+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:44.950317+0000","last_peered":"2024-09-15T21:14:44.950317+0000","last_clean":"2024-09-15T21:14:44.950317+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:44.950317+0000","last_undegraded":"2024-09-15T21:14:44.950317+0000","last_fullsized":"2024-09-15T21:14:44.950317+0000","mapping_epoch":36,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2834432,"num_objects":346,"num_object_clones":0,"num_object_copies":1384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12097","reported_seq":12165,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.004045+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:46.004045+0000","last_peered":"2024-09-15T21:14:46.004045+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:46.004045+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2741516,"num_objects":335,"num_object_clones":0,"num_object_copies":1340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1944}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12322","reported_seq":12416,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:46.725391+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:46.725391+0000","last_peered":"2024-09-15T21:14:46.725391+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:46.725391+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2938124,"num_objects":359,"num_object_clones":0,"num_object_copies":1436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12320,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1922}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12226","reported_seq":13023,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.725024+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:48.725024+0000","last_peered":"2024-09-15T21:14:48.725024+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:48.725024+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":43624348,"num_objects":5330,"num_object_clones":0,"num_object_copies":21320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168010,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":87629824,"data_stored":87629824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160892,"ondisk_log_size":160892,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":703956,"kb_used_data":22532,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93667884,"statfs":{"total":96636764160,"available":95915913216,"internally_reserved":0,"allocated":23072768,"data_stored":22891479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1,4,7,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707424,"kb_used_data":22352,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93664416,"statfs":{"total":96636764160,"available":95912361984,"internally_reserved":0,"allocated":22888448,"data_stored":22707159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051713,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564528,"kb_used_data":23392,"kb_used_omap":4,"kb_used_meta":541115,"kb_avail":93807312,"statfs":{"total":96636764160,"available":96058687488,"internally_reserved":0,"allocated":23953408,"data_stored":23772119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":554102479},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,3,2,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728152,"kb_used_data":20404,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93643688,"statfs":{"total":96636764160,"available":95891136512,"internally_reserved":0,"allocated":20893696,"data_stored":20712407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":724692686},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20291584,"data_stored":20291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23232512,"data_stored":23232512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21827584,"data_stored":21827584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":22278144,"data_stored":22278144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:51.910 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:52.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:52.518 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:52.537 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-15T21:14:52.202573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37644188,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168740,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705016,"kb_used_data":77144,"kb_used_omap":14,"kb_used_meta":2627761,"kb_avail":374782344,"statfs":{"total":386547056640,"available":383777120256,"internally_reserved":0,"allocated":78995456,"data_stored":78270300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2690827826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,7,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21282816,"num_objects":-2598,"num_object_clones":0,"num_object_copies":-10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005108"},"pg_stats":[{"pgid":"1.f","version":"37'12083","reported_seq":12139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:50.997844+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:50.997844+0000","last_peered":"2024-09-15T21:14:50.997844+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:50.997844+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12258","reported_seq":17155,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:50.060261+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:50.060261+0000","last_peered":"2024-09-15T21:14:50.060261+0000","last_clean":"2024-09-15T21:14:50.060261+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:50.060261+0000","last_undegraded":"2024-09-15T21:14:50.060261+0000","last_fullsized":"2024-09-15T21:14:50.060261+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2392064,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12125","reported_seq":12216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.709132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:51.709132+0000","last_peered":"2024-09-15T21:14:51.709132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:51.709132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2513566,"num_objects":308,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12124,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12139","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.559045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:48.559045+0000","last_peered":"2024-09-15T21:14:48.559045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:48.559045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2594060,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12410","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.189004+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:51.189004+0000","last_peered":"2024-09-15T21:14:51.189004+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:51.189004+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2037004,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12103","reported_seq":12159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.066230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:51.066230+0000","last_peered":"2024-09-15T21:14:51.066230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:51.066230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2236416,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12479","reported_seq":12560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.690006+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:48.690006+0000","last_peered":"2024-09-15T21:14:48.690006+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:48.690006+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2397452,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12430","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.601178+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:48.601178+0000","last_peered":"2024-09-15T21:14:48.601178+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:48.601178+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2400","ondisk_log_start":"27'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708748,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12164","reported_seq":12233,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.233668+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:51.233668+0000","last_peered":"2024-09-15T21:14:51.233668+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:51.233668+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353688,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12308","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.557047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:51.557047+0000","last_peered":"2024-09-15T21:14:51.557047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:51.557047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2077964,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12306,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:51.144023+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:51.144023+0000","last_peered":"2024-09-15T21:14:51.144023+0000","last_clean":"2024-09-15T21:14:51.144023+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:51.144023+0000","last_undegraded":"2024-09-15T21:14:51.144023+0000","last_fullsized":"2024-09-15T21:14:51.144023+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12160","reported_seq":12211,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.172388+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:51.172388+0000","last_peered":"2024-09-15T21:14:51.172388+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:51.172388+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185862,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12212","reported_seq":16905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:49.944718+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:49.944718+0000","last_peered":"2024-09-15T21:14:49.944718+0000","last_clean":"2024-09-15T21:14:49.944718+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:49.944718+0000","last_undegraded":"2024-09-15T21:14:49.944718+0000","last_fullsized":"2024-09-15T21:14:49.944718+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12164","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.073438+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:51.073438+0000","last_peered":"2024-09-15T21:14:51.073438+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:51.073438+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2192652,"num_objects":268,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12389","reported_seq":12483,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.593771+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:51.593771+0000","last_peered":"2024-09-15T21:14:51.593771+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:51.593771+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389260,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12387,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12226","reported_seq":13023,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.725024+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:48.725024+0000","last_peered":"2024-09-15T21:14:48.725024+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:48.725024+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":37644188,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168740,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":75460608,"data_stored":75460608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704276,"kb_used_data":18688,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93667564,"statfs":{"total":96636764160,"available":95915585536,"internally_reserved":0,"allocated":19136512,"data_stored":18955223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707656,"kb_used_data":18420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93664184,"statfs":{"total":96636764160,"available":95912124416,"internally_reserved":0,"allocated":18862080,"data_stored":18680791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564932,"kb_used_data":19632,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93806908,"statfs":{"total":96636764160,"available":96058273792,"internally_reserved":0,"allocated":20103168,"data_stored":19921879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728152,"kb_used_data":20404,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93643688,"statfs":{"total":96636764160,"available":95891136512,"internally_reserved":0,"allocated":20893696,"data_stored":20712407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":724692686},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20291584,"data_stored":20291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19177472,"data_stored":19177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17780736,"data_stored":17780736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18210816,"data_stored":18210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:52.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:52.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12083", 'reported_seq': 12139, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:50.997844+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:50.997844+0000', 'last_peered': '2024-09-15T21:14:50.997844+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:50.997844+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'2000", 'ondisk_log_start': "27'2000", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10083, 'log_dups_size': 0, 'ondisk_log_size': 10083, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2203648, 'num_objects': 269, 'num_object_clones': 0, 'num_object_copies': 1076, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 269, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12083, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2007}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.538 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:52.538 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12125", 'reported_seq': 12216, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.709132+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:51.709132+0000', 'last_peered': '2024-09-15T21:14:51.709132+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:51.709132+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10025, 'log_dups_size': 0, 'ondisk_log_size': 10025, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2513566, 'num_objects': 308, 'num_object_clones': 0, 'num_object_copies': 1232, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 308, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12124, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1950}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12139", 'reported_seq': 12219, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:48.559045+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:48.559045+0000', 'last_peered': '2024-09-15T21:14:48.559045+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:48.559045+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2594060, 'num_objects': 317, 'num_object_clones': 0, 'num_object_copies': 1268, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 317, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12137, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12410", 'reported_seq': 12477, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.189004+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:51.189004+0000', 'last_peered': '2024-09-15T21:14:51.189004+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:51.189004+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10010, 'log_dups_size': 0, 'ondisk_log_size': 10010, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2037004, 'num_objects': 249, 'num_object_clones': 0, 'num_object_copies': 996, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 249, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12408, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2062}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12103", 'reported_seq': 12159, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.066230+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:51.066230+0000', 'last_peered': '2024-09-15T21:14:51.066230+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:51.066230+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10003, 'log_dups_size': 0, 'ondisk_log_size': 10003, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2236416, 'num_objects': 273, 'num_object_clones': 0, 'num_object_copies': 1092, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 273, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12103, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2043}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.539 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:52.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12479", 'reported_seq': 12560, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:48.690006+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:48.690006+0000', 'last_peered': '2024-09-15T21:14:48.690006+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:48.690006+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2397452, 'num_objects': 293, 'num_object_clones': 0, 'num_object_copies': 1172, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 293, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12477, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2073}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:52.540 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:52.540 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12430", 'reported_seq': 12508, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:48.601178+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:48.601178+0000', 'last_peered': '2024-09-15T21:14:48.601178+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:48.601178+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "27'2400", 'ondisk_log_start': "27'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2708748, 'num_objects': 331, 'num_object_clones': 0, 'num_object_copies': 1324, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 331, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2143}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12164", 'reported_seq': 12233, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.233668+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:51.233668+0000', 'last_peered': '2024-09-15T21:14:51.233668+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:51.233668+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2353688, 'num_objects': 288, 'num_object_clones': 0, 'num_object_copies': 1152, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 288, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12161, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2044}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12308", 'reported_seq': 12400, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.557047+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:51.557047+0000', 'last_peered': '2024-09-15T21:14:51.557047+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:51.557047+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10008, 'log_dups_size': 0, 'ondisk_log_size': 10008, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2077964, 'num_objects': 254, 'num_object_clones': 0, 'num_object_copies': 1016, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 254, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12306, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2081}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12160", 'reported_seq': 12211, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.172388+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:51.172388+0000', 'last_peered': '2024-09-15T21:14:51.172388+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:51.172388+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2185862, 'num_objects': 267, 'num_object_clones': 0, 'num_object_copies': 1068, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 267, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12159, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.541 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12164", 'reported_seq': 12232, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.073438+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:51.073438+0000', 'last_peered': '2024-09-15T21:14:51.073438+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:51.073438+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10064, 'log_dups_size': 0, 'ondisk_log_size': 10064, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2192652, 'num_objects': 268, 'num_object_clones': 0, 'num_object_copies': 1072, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 268, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12162, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2011}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12389", 'reported_seq': 12483, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:51.593771+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:51.593771+0000', 'last_peered': '2024-09-15T21:14:51.593771+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:51.593771+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10089, 'log_dups_size': 0, 'ondisk_log_size': 10089, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2389260, 'num_objects': 292, 'num_object_clones': 0, 'num_object_copies': 1168, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 292, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12387, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1989}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:52.542 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12226", 'reported_seq': 13023, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:48.725024+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:48.725024+0000', 'last_peered': '2024-09-15T21:14:48.725024+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:48.725024+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2643212, 'num_objects': 323, 'num_object_clones': 0, 'num_object_copies': 1292, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1959, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 323, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12224, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 360, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1636}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:52.543 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:53.323 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:53.323 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29796/18400 objects degraded (161.935%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":759,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4600,"data_bytes":37644188,"bytes_used":2769936384,"bytes_avail":383777120256,"bytes_total":386547056640,"degraded_objects":29796,"degraded_total":18400,"degraded_ratio":1.6193478260869565,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":216},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:53.349 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:53.349 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:53.996 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:53.997 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:54.028 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-15T21:14:52.202573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":37644188,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168740,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2705016,"kb_used_data":77144,"kb_used_omap":14,"kb_used_meta":2627761,"kb_avail":374782344,"statfs":{"total":386547056640,"available":383777120256,"internally_reserved":0,"allocated":78995456,"data_stored":78270300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2690827826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,3,7,7,10,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21282816,"num_objects":-2598,"num_object_clones":0,"num_object_copies":-10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2598,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005108"},"pg_stats":[{"pgid":"1.f","version":"37'12083","reported_seq":12139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:50.997844+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:50.997844+0000","last_peered":"2024-09-15T21:14:50.997844+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:50.997844+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12258","reported_seq":17155,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:50.060261+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:50.060261+0000","last_peered":"2024-09-15T21:14:50.060261+0000","last_clean":"2024-09-15T21:14:50.060261+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:50.060261+0000","last_undegraded":"2024-09-15T21:14:50.060261+0000","last_fullsized":"2024-09-15T21:14:50.060261+0000","mapping_epoch":34,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2392064,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12125","reported_seq":12216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.709132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:51.709132+0000","last_peered":"2024-09-15T21:14:51.709132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:51.709132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2513566,"num_objects":308,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12124,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12139","reported_seq":12219,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.559045+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:48.559045+0000","last_peered":"2024-09-15T21:14:48.559045+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:48.559045+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2594060,"num_objects":317,"num_object_clones":0,"num_object_copies":1268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2017}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12410","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.189004+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:51.189004+0000","last_peered":"2024-09-15T21:14:51.189004+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:51.189004+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2037004,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12103","reported_seq":12159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.066230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:51.066230+0000","last_peered":"2024-09-15T21:14:51.066230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:51.066230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2236416,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12479","reported_seq":12560,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.690006+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:48.690006+0000","last_peered":"2024-09-15T21:14:48.690006+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:48.690006+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2397452,"num_objects":293,"num_object_clones":0,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12430","reported_seq":12508,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.601178+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:48.601178+0000","last_peered":"2024-09-15T21:14:48.601178+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:48.601178+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"27'2400","ondisk_log_start":"27'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2708748,"num_objects":331,"num_object_clones":0,"num_object_copies":1324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2143}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12164","reported_seq":12233,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.233668+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:51.233668+0000","last_peered":"2024-09-15T21:14:51.233668+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:51.233668+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353688,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12308","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.557047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:51.557047+0000","last_peered":"2024-09-15T21:14:51.557047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:51.557047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2077964,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12306,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:51.144023+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:51.144023+0000","last_peered":"2024-09-15T21:14:51.144023+0000","last_clean":"2024-09-15T21:14:51.144023+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:51.144023+0000","last_undegraded":"2024-09-15T21:14:51.144023+0000","last_fullsized":"2024-09-15T21:14:51.144023+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12160","reported_seq":12211,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.172388+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:51.172388+0000","last_peered":"2024-09-15T21:14:51.172388+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:51.172388+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185862,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12212","reported_seq":16905,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:49.944718+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:49.944718+0000","last_peered":"2024-09-15T21:14:49.944718+0000","last_clean":"2024-09-15T21:14:49.944718+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:49.944718+0000","last_undegraded":"2024-09-15T21:14:49.944718+0000","last_fullsized":"2024-09-15T21:14:49.944718+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12164","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.073438+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:51.073438+0000","last_peered":"2024-09-15T21:14:51.073438+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:51.073438+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2192652,"num_objects":268,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12389","reported_seq":12483,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.593771+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:51.593771+0000","last_peered":"2024-09-15T21:14:51.593771+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:51.593771+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389260,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12387,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12226","reported_seq":13023,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:48.725024+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:48.725024+0000","last_peered":"2024-09-15T21:14:48.725024+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:48.725024+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2643212,"num_objects":323,"num_object_clones":0,"num_object_copies":1292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1959,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12224,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":360,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1636}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":37644188,"num_objects":4600,"num_object_clones":0,"num_object_copies":18400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168740,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":75460608,"data_stored":75460608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160722,"ondisk_log_size":160722,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704276,"kb_used_data":18688,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93667564,"statfs":{"total":96636764160,"available":95915585536,"internally_reserved":0,"allocated":19136512,"data_stored":18955223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707656,"kb_used_data":18420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93664184,"statfs":{"total":96636764160,"available":95912124416,"internally_reserved":0,"allocated":18862080,"data_stored":18680791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051714,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564932,"kb_used_data":19632,"kb_used_omap":4,"kb_used_meta":545275,"kb_avail":93806908,"statfs":{"total":96636764160,"available":96058273792,"internally_reserved":0,"allocated":20103168,"data_stored":19921879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":558362319},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,4,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":728152,"kb_used_data":20404,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93643688,"statfs":{"total":96636764160,"available":95891136512,"internally_reserved":0,"allocated":20893696,"data_stored":20712407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":724692686},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,5,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20291584,"data_stored":20291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19177472,"data_stored":19177472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17780736,"data_stored":17780736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18210816,"data_stored":18210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:57.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:57.600 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:57.600 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:57.625 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-15T21:14:56.204311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":34523036,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169121,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2672152,"kb_used_data":69252,"kb_used_omap":14,"kb_used_meta":2602801,"kb_avail":374815208,"statfs":{"total":386547056640,"available":383810772992,"internally_reserved":0,"allocated":70914048,"data_stored":70188892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2665268786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,2,4,8,6,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17170432,"num_objects":-2096,"num_object_clones":0,"num_object_copies":-8384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005105"},"pg_stats":[{"pgid":"1.f","version":"37'12083","reported_seq":12139,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:50.997844+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:50.997844+0000","last_peered":"2024-09-15T21:14:50.997844+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:50.997844+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2203648,"num_objects":269,"num_object_clones":0,"num_object_copies":1076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":269,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2007}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12321","reported_seq":17219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.312465+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:55.312465+0000","last_peered":"2024-09-15T21:14:55.312465+0000","last_clean":"2024-09-15T21:14:55.312465+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:55.312465+0000","last_undegraded":"2024-09-15T21:14:55.312465+0000","last_fullsized":"2024-09-15T21:14:55.312465+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1875968,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12125","reported_seq":12216,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.709132+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:51.709132+0000","last_peered":"2024-09-15T21:14:51.709132+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:51.709132+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2513566,"num_objects":308,"num_object_clones":0,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12124,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1950}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12199","reported_seq":12279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.423580+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:53.423580+0000","last_peered":"2024-09-15T21:14:53.423580+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:53.423580+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12410","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.189004+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:51.189004+0000","last_peered":"2024-09-15T21:14:51.189004+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:51.189004+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2037004,"num_objects":249,"num_object_clones":0,"num_object_copies":996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2062}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12103","reported_seq":12159,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.066230+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:51.066230+0000","last_peered":"2024-09-15T21:14:51.066230+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:51.066230+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2236416,"num_objects":273,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2043}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12546","reported_seq":12627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.527034+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:53.527034+0000","last_peered":"2024-09-15T21:14:53.527034+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:53.527034+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848588,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12501","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.782630+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:53.782630+0000","last_peered":"2024-09-15T21:14:53.782630+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:53.782630+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2214}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12164","reported_seq":12233,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.233668+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:51.233668+0000","last_peered":"2024-09-15T21:14:51.233668+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:51.233668+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2353688,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2044}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12308","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.557047+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:51.557047+0000","last_peered":"2024-09-15T21:14:51.557047+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:51.557047+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2077964,"num_objects":254,"num_object_clones":0,"num_object_copies":1016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12306,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2081}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12072","reported_seq":16745,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:51.144023+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:51.144023+0000","last_peered":"2024-09-15T21:14:51.144023+0000","last_clean":"2024-09-15T21:14:51.144023+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:51.144023+0000","last_undegraded":"2024-09-15T21:14:51.144023+0000","last_fullsized":"2024-09-15T21:14:51.144023+0000","mapping_epoch":32,"log_start":"27'2000","ondisk_log_start":"27'2000","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":288,"num_object_clones":0,"num_object_copies":1152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12160","reported_seq":12211,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.172388+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:51.172388+0000","last_peered":"2024-09-15T21:14:51.172388+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:51.172388+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2185862,"num_objects":267,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12159,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12266","reported_seq":16960,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.206145+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:55.206145+0000","last_peered":"2024-09-15T21:14:55.206145+0000","last_clean":"2024-09-15T21:14:55.206145+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:55.206145+0000","last_undegraded":"2024-09-15T21:14:55.206145+0000","last_fullsized":"2024-09-15T21:14:55.206145+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916928,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12164","reported_seq":12232,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.073438+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:51.073438+0000","last_peered":"2024-09-15T21:14:51.073438+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:51.073438+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2192652,"num_objects":268,"num_object_clones":0,"num_object_copies":1072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2011}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12389","reported_seq":12483,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:51.593771+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:51.593771+0000","last_peered":"2024-09-15T21:14:51.593771+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:51.593771+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2389260,"num_objects":292,"num_object_clones":0,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12387,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1989}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12292","reported_seq":13162,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.766616+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:53.766616+0000","last_peered":"2024-09-15T21:14:53.766616+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:53.766616+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":34523036,"num_objects":4219,"num_object_clones":0,"num_object_copies":16876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4219,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169121,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":67395584,"data_stored":67395584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160803,"ondisk_log_size":160803,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704276,"kb_used_data":18688,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93667564,"statfs":{"total":96636764160,"available":95915585536,"internally_reserved":0,"allocated":19136512,"data_stored":18955223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":707656,"kb_used_data":18420,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93664184,"statfs":{"total":96636764160,"available":95912124416,"internally_reserved":0,"allocated":18862080,"data_stored":18680791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548436,"kb_used_data":15612,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93823404,"statfs":{"total":96636764160,"available":96075165696,"internally_reserved":0,"allocated":15986688,"data_stored":15805399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711784,"kb_used_data":16532,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93660056,"statfs":{"total":96636764160,"available":95907897344,"internally_reserved":0,"allocated":16928768,"data_stored":16747479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16289792,"data_stored":16289792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":17780736,"data_stored":17780736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":18210816,"data_stored":18210816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:57.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:58.255 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:58.255 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:58.292 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-15T21:14:58.205177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29558684,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169727,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656176,"kb_used_data":61580,"kb_used_omap":14,"kb_used_meta":2594481,"kb_avail":374831184,"statfs":{"total":386547056640,"available":383827132416,"internally_reserved":0,"allocated":63057920,"data_stored":62332764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2656749106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,2,12,9,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17743872,"num_objects":-2166,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2166,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005082"},"pg_stats":[{"pgid":"1.f","version":"37'12146","reported_seq":12202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.334259+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:56.334259+0000","last_peered":"2024-09-15T21:14:56.334259+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:56.334259+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1687552,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2070}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12321","reported_seq":17219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.312465+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:55.312465+0000","last_peered":"2024-09-15T21:14:55.312465+0000","last_clean":"2024-09-15T21:14:55.312465+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:55.312465+0000","last_undegraded":"2024-09-15T21:14:55.312465+0000","last_fullsized":"2024-09-15T21:14:55.312465+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1875968,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12196","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.780948+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:56.780948+0000","last_peered":"2024-09-15T21:14:56.780948+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:56.780948+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931934,"num_objects":237,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12195,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12199","reported_seq":12279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.423580+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:53.423580+0000","last_peered":"2024-09-15T21:14:53.423580+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:53.423580+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12465","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.149156+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:56.149156+0000","last_peered":"2024-09-15T21:14:56.149156+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:56.149156+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1586444,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12165","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.270012+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:56.270012+0000","last_peered":"2024-09-15T21:14:56.270012+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:56.270012+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1728512,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12546","reported_seq":12627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.527034+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:53.527034+0000","last_peered":"2024-09-15T21:14:53.527034+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:53.527034+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848588,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12501","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.782630+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:53.782630+0000","last_peered":"2024-09-15T21:14:53.782630+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:53.782630+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2214}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12220","reported_seq":12291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.355375+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:56.355375+0000","last_peered":"2024-09-15T21:14:56.355375+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:56.355375+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1894936,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12367","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.653152+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:56.653152+0000","last_peered":"2024-09-15T21:14:56.653152+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:56.653152+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594636,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12126","reported_seq":16797,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:56.255570+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:56.255570+0000","last_peered":"2024-09-15T21:14:56.255570+0000","last_clean":"2024-09-15T21:14:56.255570+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:56.255570+0000","last_undegraded":"2024-09-15T21:14:56.255570+0000","last_fullsized":"2024-09-15T21:14:56.255570+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916928,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6237,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12220","reported_seq":12271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.195964+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:56.195964+0000","last_peered":"2024-09-15T21:14:56.195964+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:56.195964+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1694342,"num_objects":207,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2049}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12266","reported_seq":16960,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.206145+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:55.206145+0000","last_peered":"2024-09-15T21:14:55.206145+0000","last_clean":"2024-09-15T21:14:55.206145+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:55.206145+0000","last_undegraded":"2024-09-15T21:14:55.206145+0000","last_fullsized":"2024-09-15T21:14:55.206145+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916928,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12227","reported_seq":12295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.293349+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:56.293349+0000","last_peered":"2024-09-15T21:14:56.293349+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:56.293349+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676556,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12452","reported_seq":12546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.580891+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:56.580891+0000","last_peered":"2024-09-15T21:14:56.580891+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:56.580891+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1873164,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2052}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12292","reported_seq":13162,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.766616+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:53.766616+0000","last_peered":"2024-09-15T21:14:53.766616+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:53.766616+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":29558684,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169727,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":59363328,"data_stored":59363328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704628,"kb_used_data":14880,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93667212,"statfs":{"total":96636764160,"available":95915225088,"internally_reserved":0,"allocated":15237120,"data_stored":15055831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691328,"kb_used_data":14556,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93680512,"statfs":{"total":96636764160,"available":95928844288,"internally_reserved":0,"allocated":14905344,"data_stored":14724055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548436,"kb_used_data":15612,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93823404,"statfs":{"total":96636764160,"available":96075165696,"internally_reserved":0,"allocated":15986688,"data_stored":15805399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711784,"kb_used_data":16532,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93660056,"statfs":{"total":96636764160,"available":95907897344,"internally_reserved":0,"allocated":16928768,"data_stored":16747479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16289792,"data_stored":16289792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13803520,"data_stored":13803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14155776,"data_stored":14155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12146", 'reported_seq': 12202, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.334259+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:14:56.334259+0000', 'last_peered': '2024-09-15T21:14:56.334259+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:14:56.334259+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1687552, 'num_objects': 206, 'num_object_clones': 0, 'num_object_copies': 824, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 206, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12146, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2070}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12196", 'reported_seq': 12287, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.780948+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:14:56.780948+0000', 'last_peered': '2024-09-15T21:14:56.780948+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:14:56.780948+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10096, 'log_dups_size': 0, 'ondisk_log_size': 10096, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1931934, 'num_objects': 237, 'num_object_clones': 0, 'num_object_copies': 948, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 237, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12195, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2021}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12199", 'reported_seq': 12279, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:53.423580+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:53.423580+0000', 'last_peered': '2024-09-15T21:14:53.423580+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:53.423580+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2102540, 'num_objects': 257, 'num_object_clones': 0, 'num_object_copies': 1028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12197, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2077}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:14:58.294 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12465", 'reported_seq': 12532, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.149156+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:14:56.149156+0000', 'last_peered': '2024-09-15T21:14:56.149156+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:14:56.149156+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1586444, 'num_objects': 194, 'num_object_clones': 0, 'num_object_copies': 776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12463, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2117}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:14:58.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12165", 'reported_seq': 12221, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.270012+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:14:56.270012+0000', 'last_peered': '2024-09-15T21:14:56.270012+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:14:56.270012+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10065, 'log_dups_size': 0, 'ondisk_log_size': 10065, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1728512, 'num_objects': 211, 'num_object_clones': 0, 'num_object_copies': 844, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 211, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12165, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2105}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.295 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:14:58.295 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12546", 'reported_seq': 12627, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:53.527034+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:53.527034+0000', 'last_peered': '2024-09-15T21:14:53.527034+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:53.527034+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10046, 'log_dups_size': 0, 'ondisk_log_size': 10046, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1848588, 'num_objects': 226, 'num_object_clones': 0, 'num_object_copies': 904, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 226, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12544, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2140}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:58.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:14:58.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12501", 'reported_seq': 12581, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:53.782630+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:53.782630+0000', 'last_peered': '2024-09-15T21:14:53.782630+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:53.782630+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2127116, 'num_objects': 260, 'num_object_clones': 0, 'num_object_copies': 1040, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 260, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12499, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2214}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:58.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:14:58.296 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12220", 'reported_seq': 12291, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.355375+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:14:56.355375+0000', 'last_peered': '2024-09-15T21:14:56.355375+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:14:56.355375+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1894936, 'num_objects': 232, 'num_object_clones': 0, 'num_object_copies': 928, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 232, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12217, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2100}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.296 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:14:58.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12367", 'reported_seq': 12459, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.653152+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:14:56.653152+0000', 'last_peered': '2024-09-15T21:14:56.653152+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:14:56.653152+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1594636, 'num_objects': 195, 'num_object_clones': 0, 'num_object_copies': 780, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 195, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12365, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2140}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:58.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:14:58.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12220", 'reported_seq': 12271, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.195964+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:14:56.195964+0000', 'last_peered': '2024-09-15T21:14:56.195964+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:14:56.195964+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1694342, 'num_objects': 207, 'num_object_clones': 0, 'num_object_copies': 828, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 207, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12219, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2049}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.297 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:14:58.297 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12227", 'reported_seq': 12295, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.293349+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:14:56.293349+0000', 'last_peered': '2024-09-15T21:14:56.293349+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:14:56.293349+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1676556, 'num_objects': 205, 'num_object_clones': 0, 'num_object_copies': 820, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 205, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12225, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2074}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:14:58.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:14:58.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12452", 'reported_seq': 12546, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:56.580891+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:14:56.580891+0000', 'last_peered': '2024-09-15T21:14:56.580891+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:14:56.580891+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1873164, 'num_objects': 229, 'num_object_clones': 0, 'num_object_copies': 916, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 229, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12450, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2052}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:14:58.298 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:14:58.298 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12292", 'reported_seq': 13162, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:53.766616+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:53.766616+0000', 'last_peered': '2024-09-15T21:14:53.766616+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:53.766616+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10092, 'log_dups_size': 0, 'ondisk_log_size': 10092, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2102540, 'num_objects': 257, 'num_object_clones': 0, 'num_object_copies': 1028, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1922, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 257, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12290, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 397, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1665}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:14:58.299 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:14:59.052 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:59.052 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29759/14452 objects degraded (205.916%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":764,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":3613,"data_bytes":29558684,"bytes_used":2719924224,"bytes_avail":383827132416,"bytes_total":386547056640,"degraded_objects":29759,"degraded_total":14452,"degraded_ratio":2.059161361749239,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:14:59.080 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:14:59.080 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:14:59.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:14:59.660 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:14:59.687 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-15T21:14:58.205177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":29558684,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169727,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2656176,"kb_used_data":61580,"kb_used_omap":14,"kb_used_meta":2594481,"kb_avail":374831184,"statfs":{"total":386547056640,"available":383827132416,"internally_reserved":0,"allocated":63057920,"data_stored":62332764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2656749106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,2,12,9,7,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17743872,"num_objects":-2166,"num_object_clones":0,"num_object_copies":-8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-79,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2166,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005082"},"pg_stats":[{"pgid":"1.f","version":"37'12146","reported_seq":12202,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.334259+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:14:56.334259+0000","last_peered":"2024-09-15T21:14:56.334259+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:14:56.334259+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1687552,"num_objects":206,"num_object_clones":0,"num_object_copies":824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12146,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2070}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12321","reported_seq":17219,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.312465+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:14:55.312465+0000","last_peered":"2024-09-15T21:14:55.312465+0000","last_clean":"2024-09-15T21:14:55.312465+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:14:55.312465+0000","last_undegraded":"2024-09-15T21:14:55.312465+0000","last_fullsized":"2024-09-15T21:14:55.312465+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1875968,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2987,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12196","reported_seq":12287,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.780948+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:14:56.780948+0000","last_peered":"2024-09-15T21:14:56.780948+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:14:56.780948+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1931934,"num_objects":237,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12195,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2021}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12199","reported_seq":12279,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.423580+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:53.423580+0000","last_peered":"2024-09-15T21:14:53.423580+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:53.423580+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2077}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12465","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.149156+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:14:56.149156+0000","last_peered":"2024-09-15T21:14:56.149156+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:14:56.149156+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1586444,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12463,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2117}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12165","reported_seq":12221,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.270012+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:14:56.270012+0000","last_peered":"2024-09-15T21:14:56.270012+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:14:56.270012+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1728512,"num_objects":211,"num_object_clones":0,"num_object_copies":844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2105}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12546","reported_seq":12627,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.527034+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:53.527034+0000","last_peered":"2024-09-15T21:14:53.527034+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:53.527034+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1848588,"num_objects":226,"num_object_clones":0,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2140}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12501","reported_seq":12581,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.782630+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:53.782630+0000","last_peered":"2024-09-15T21:14:53.782630+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:53.782630+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2127116,"num_objects":260,"num_object_clones":0,"num_object_copies":1040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12499,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2214}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12220","reported_seq":12291,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.355375+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:14:56.355375+0000","last_peered":"2024-09-15T21:14:56.355375+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:14:56.355375+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1894936,"num_objects":232,"num_object_clones":0,"num_object_copies":928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12217,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2100}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12367","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.653152+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:14:56.653152+0000","last_peered":"2024-09-15T21:14:56.653152+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:14:56.653152+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1594636,"num_objects":195,"num_object_clones":0,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2140}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12126","reported_seq":16797,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:56.255570+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:14:56.255570+0000","last_peered":"2024-09-15T21:14:56.255570+0000","last_clean":"2024-09-15T21:14:56.255570+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:14:56.255570+0000","last_undegraded":"2024-09-15T21:14:56.255570+0000","last_fullsized":"2024-09-15T21:14:56.255570+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916928,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6237,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12220","reported_seq":12271,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.195964+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:14:56.195964+0000","last_peered":"2024-09-15T21:14:56.195964+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:14:56.195964+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1694342,"num_objects":207,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2049}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12266","reported_seq":16960,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:14:55.206145+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:14:55.206145+0000","last_peered":"2024-09-15T21:14:55.206145+0000","last_clean":"2024-09-15T21:14:55.206145+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:14:55.206145+0000","last_undegraded":"2024-09-15T21:14:55.206145+0000","last_fullsized":"2024-09-15T21:14:55.206145+0000","mapping_epoch":36,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1916928,"num_objects":234,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":528,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12227","reported_seq":12295,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.293349+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:14:56.293349+0000","last_peered":"2024-09-15T21:14:56.293349+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:14:56.293349+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1676556,"num_objects":205,"num_object_clones":0,"num_object_copies":820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12225,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2074}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12452","reported_seq":12546,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:56.580891+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:14:56.580891+0000","last_peered":"2024-09-15T21:14:56.580891+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:14:56.580891+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1873164,"num_objects":229,"num_object_clones":0,"num_object_copies":916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2052}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12292","reported_seq":13162,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:53.766616+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:53.766616+0000","last_peered":"2024-09-15T21:14:53.766616+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:53.766616+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2102540,"num_objects":257,"num_object_clones":0,"num_object_copies":1028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12290,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1665}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":29558684,"num_objects":3613,"num_object_clones":0,"num_object_copies":14452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169727,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":59363328,"data_stored":59363328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160809,"ondisk_log_size":160809,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704628,"kb_used_data":14880,"kb_used_omap":4,"kb_used_meta":689723,"kb_avail":93667212,"statfs":{"total":96636764160,"available":95915225088,"internally_reserved":0,"allocated":15237120,"data_stored":15055831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":706277072},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,4,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691328,"kb_used_data":14556,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":93680512,"statfs":{"total":96636764160,"available":95928844288,"internally_reserved":0,"allocated":14905344,"data_stored":14724055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,1,6,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051715,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":548436,"kb_used_data":15612,"kb_used_omap":4,"kb_used_meta":532795,"kb_avail":93823404,"statfs":{"total":96636764160,"available":96075165696,"internally_reserved":0,"allocated":15986688,"data_stored":15805399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":545582799},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247953,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":711784,"kb_used_data":16532,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93660056,"statfs":{"total":96636764160,"available":95907897344,"internally_reserved":0,"allocated":16928768,"data_stored":16747479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16289792,"data_stored":16289792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15114240,"data_stored":15114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13803520,"data_stored":13803520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14155776,"data_stored":14155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:02.689 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:03.339 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:03.340 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:03.368 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-15T21:15:02.206929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21522332,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170708,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658192,"kb_used_data":45564,"kb_used_omap":14,"kb_used_meta":2612529,"kb_avail":374829168,"statfs":{"total":386547056640,"available":383825068032,"internally_reserved":0,"allocated":46657536,"data_stored":45932380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2675230258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,5,19,20,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22102016,"num_objects":-2698,"num_object_clones":0,"num_object_copies":-10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"37'12199","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.390170+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:01.390170+0000","last_peered":"2024-09-15T21:15:01.390170+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:01.390170+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12380","reported_seq":17278,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.357235+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:00.357235+0000","last_peered":"2024-09-15T21:15:00.357235+0000","last_clean":"2024-09-15T21:15:00.357235+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:00.357235+0000","last_undegraded":"2024-09-15T21:15:00.357235+0000","last_fullsized":"2024-09-15T21:15:00.357235+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1392640,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12270","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.611021+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:01.611021+0000","last_peered":"2024-09-15T21:15:01.611021+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:01.611021+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325726,"num_objects":163,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12269,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12260","reported_seq":12340,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.840464+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:58.840464+0000","last_peered":"2024-09-15T21:14:58.840464+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:58.840464+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1602828,"num_objects":196,"num_object_clones":0,"num_object_copies":784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12520","reported_seq":12587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.451175+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:01.451175+0000","last_peered":"2024-09-15T21:15:01.451175+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:01.451175+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12227","reported_seq":12283,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.400983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:01.400983+0000","last_peered":"2024-09-15T21:15:01.400983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:01.400983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1220608,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12227,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12600","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.763295+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:58.763295+0000","last_peered":"2024-09-15T21:14:58.763295+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:58.763295+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1406220,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12598,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2194}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12567","reported_seq":12647,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.663897+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:58.663897+0000","last_peered":"2024-09-15T21:14:58.663897+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:58.663897+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1586444,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12281","reported_seq":12352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.395686+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:01.395686+0000","last_peered":"2024-09-15T21:15:01.395686+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:01.395686+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1395224,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12421","reported_seq":12513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.765398+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:01.765398+0000","last_peered":"2024-09-15T21:15:01.765398+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:01.765398+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152268,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12196","reported_seq":16870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:01.492382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:01.492382+0000","last_peered":"2024-09-15T21:15:01.492382+0000","last_clean":"2024-09-15T21:15:01.492382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:01.492382+0000","last_undegraded":"2024-09-15T21:15:01.492382+0000","last_fullsized":"2024-09-15T21:15:01.492382+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1343488,"num_objects":164,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12282","reported_seq":12332,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.377055+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:01.377055+0000","last_peered":"2024-09-15T21:15:01.377055+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:01.377055+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1186438,"num_objects":145,"num_object_clones":0,"num_object_copies":580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12334","reported_seq":17028,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.178190+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:00.178190+0000","last_peered":"2024-09-15T21:15:00.178190+0000","last_clean":"2024-09-15T21:15:00.178190+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:00.178190+0000","last_undegraded":"2024-09-15T21:15:00.178190+0000","last_fullsized":"2024-09-15T21:15:00.178190+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1359872,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":596,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12279","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.486734+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:01.486734+0000","last_peered":"2024-09-15T21:15:01.486734+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:01.486734+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250572,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12515","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.831618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:01.831618+0000","last_peered":"2024-09-15T21:15:01.831618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:01.831618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1357068,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12359","reported_seq":13310,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.818041+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:58.818041+0000","last_peered":"2024-09-15T21:14:58.818041+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:58.818041+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1553676,"num_objects":190,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1692}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":21522332,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170708,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43462656,"data_stored":43462656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687612,"kb_used_data":10344,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93684228,"statfs":{"total":96636764160,"available":95932649472,"internally_reserved":0,"allocated":10592256,"data_stored":10410967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,10,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691668,"kb_used_data":10756,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93680172,"statfs":{"total":96636764160,"available":95928496128,"internally_reserved":0,"allocated":11014144,"data_stored":10832855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566852,"kb_used_data":11812,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93804988,"statfs":{"total":96636764160,"available":96056307712,"internally_reserved":0,"allocated":12095488,"data_stored":11914199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,4,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712060,"kb_used_data":12652,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93659780,"statfs":{"total":96636764160,"available":95907614720,"internally_reserved":0,"allocated":12955648,"data_stored":12774359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,5,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12263424,"data_stored":12263424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11177984,"data_stored":11177984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9842688,"data_stored":9842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10178560,"data_stored":10178560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:03.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:03.937 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:03.938 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:03.965 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-15T21:15:02.206929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21522332,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170708,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658192,"kb_used_data":45564,"kb_used_omap":14,"kb_used_meta":2612529,"kb_avail":374829168,"statfs":{"total":386547056640,"available":383825068032,"internally_reserved":0,"allocated":46657536,"data_stored":45932380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2675230258},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,5,5,19,20,18,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22102016,"num_objects":-2698,"num_object_clones":0,"num_object_copies":-10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2698,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005179"},"pg_stats":[{"pgid":"1.f","version":"37'12199","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.390170+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:01.390170+0000","last_peered":"2024-09-15T21:15:01.390170+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:01.390170+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12380","reported_seq":17278,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.357235+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:00.357235+0000","last_peered":"2024-09-15T21:15:00.357235+0000","last_clean":"2024-09-15T21:15:00.357235+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:00.357235+0000","last_undegraded":"2024-09-15T21:15:00.357235+0000","last_fullsized":"2024-09-15T21:15:00.357235+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1392640,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12270","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.611021+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:01.611021+0000","last_peered":"2024-09-15T21:15:01.611021+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:01.611021+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325726,"num_objects":163,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12269,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12260","reported_seq":12340,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.840464+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:14:58.840464+0000","last_peered":"2024-09-15T21:14:58.840464+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:14:58.840464+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1602828,"num_objects":196,"num_object_clones":0,"num_object_copies":784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2138}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12520","reported_seq":12587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.451175+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:01.451175+0000","last_peered":"2024-09-15T21:15:01.451175+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:01.451175+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12227","reported_seq":12283,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.400983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:01.400983+0000","last_peered":"2024-09-15T21:15:01.400983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:01.400983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1220608,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12227,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12600","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.763295+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:14:58.763295+0000","last_peered":"2024-09-15T21:14:58.763295+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:14:58.763295+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1406220,"num_objects":172,"num_object_clones":0,"num_object_copies":688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12598,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2194}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12567","reported_seq":12647,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.663897+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:14:58.663897+0000","last_peered":"2024-09-15T21:14:58.663897+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:14:58.663897+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1586444,"num_objects":194,"num_object_clones":0,"num_object_copies":776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12565,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2280}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12281","reported_seq":12352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.395686+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:01.395686+0000","last_peered":"2024-09-15T21:15:01.395686+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:01.395686+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1395224,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12421","reported_seq":12513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.765398+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:01.765398+0000","last_peered":"2024-09-15T21:15:01.765398+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:01.765398+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152268,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12196","reported_seq":16870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:01.492382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:01.492382+0000","last_peered":"2024-09-15T21:15:01.492382+0000","last_clean":"2024-09-15T21:15:01.492382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:01.492382+0000","last_undegraded":"2024-09-15T21:15:01.492382+0000","last_fullsized":"2024-09-15T21:15:01.492382+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1343488,"num_objects":164,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12282","reported_seq":12332,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.377055+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:01.377055+0000","last_peered":"2024-09-15T21:15:01.377055+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:01.377055+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1186438,"num_objects":145,"num_object_clones":0,"num_object_copies":580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12334","reported_seq":17028,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.178190+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:00.178190+0000","last_peered":"2024-09-15T21:15:00.178190+0000","last_clean":"2024-09-15T21:15:00.178190+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:00.178190+0000","last_undegraded":"2024-09-15T21:15:00.178190+0000","last_fullsized":"2024-09-15T21:15:00.178190+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1359872,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":596,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12279","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.486734+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:01.486734+0000","last_peered":"2024-09-15T21:15:01.486734+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:01.486734+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250572,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12515","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.831618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:01.831618+0000","last_peered":"2024-09-15T21:15:01.831618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:01.831618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1357068,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12359","reported_seq":13310,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:14:58.818041+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:14:58.818041+0000","last_peered":"2024-09-15T21:14:58.818041+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:14:58.818041+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1553676,"num_objects":190,"num_object_clones":0,"num_object_copies":760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1882,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12357,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1692}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":21522332,"num_objects":2632,"num_object_clones":0,"num_object_copies":10528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2632,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170708,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43462656,"data_stored":43462656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160990,"ondisk_log_size":160990,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687612,"kb_used_data":10344,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93684228,"statfs":{"total":96636764160,"available":95932649472,"internally_reserved":0,"allocated":10592256,"data_stored":10410967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,10,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691668,"kb_used_data":10756,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93680172,"statfs":{"total":96636764160,"available":95928496128,"internally_reserved":0,"allocated":11014144,"data_stored":10832855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566852,"kb_used_data":11812,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93804988,"statfs":{"total":96636764160,"available":96056307712,"internally_reserved":0,"allocated":12095488,"data_stored":11914199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,4,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247954,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712060,"kb_used_data":12652,"kb_used_omap":4,"kb_used_meta":699387,"kb_avail":93659780,"statfs":{"total":96636764160,"available":95907614720,"internally_reserved":0,"allocated":12955648,"data_stored":12774359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":716173006},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,5,5,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12263424,"data_stored":12263424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11177984,"data_stored":11177984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9842688,"data_stored":9842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10178560,"data_stored":10178560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:03.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:03.966 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12199", 'reported_seq': 12256, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.390170+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:15:01.390170+0000', 'last_peered': '2024-09-15T21:15:01.390170+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:01.390170+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'2100", 'ondisk_log_start': "27'2100", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10099, 'log_dups_size': 0, 'ondisk_log_size': 10099, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1253376, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12199, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2123}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.966 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12270", 'reported_seq': 12361, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.611021+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:01.611021+0000', 'last_peered': '2024-09-15T21:15:01.611021+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:01.611021+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1325726, 'num_objects': 163, 'num_object_clones': 0, 'num_object_copies': 652, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 163, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12269, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2095}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12260", 'reported_seq': 12340, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:58.840464+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:14:58.840464+0000', 'last_peered': '2024-09-15T21:14:58.840464+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:14:58.840464+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1602828, 'num_objects': 196, 'num_object_clones': 0, 'num_object_copies': 784, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 196, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12258, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2138}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12520", 'reported_seq': 12587, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.451175+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:01.451175+0000', 'last_peered': '2024-09-15T21:15:01.451175+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:01.451175+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1135884, 'num_objects': 139, 'num_object_clones': 0, 'num_object_copies': 556, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 139, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12518, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2172}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12227", 'reported_seq': 12283, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.400983+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:01.400983+0000', 'last_peered': '2024-09-15T21:15:01.400983+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:01.400983+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10027, 'log_dups_size': 0, 'ondisk_log_size': 10027, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1220608, 'num_objects': 149, 'num_object_clones': 0, 'num_object_copies': 596, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 149, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12227, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2167}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12600", 'reported_seq': 12681, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:58.763295+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:14:58.763295+0000', 'last_peered': '2024-09-15T21:14:58.763295+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:14:58.763295+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10100, 'log_dups_size': 0, 'ondisk_log_size': 10100, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1406220, 'num_objects': 172, 'num_object_clones': 0, 'num_object_copies': 688, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 172, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12598, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2194}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:03.967 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12567", 'reported_seq': 12647, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:58.663897+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:14:58.663897+0000', 'last_peered': '2024-09-15T21:14:58.663897+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:14:58.663897+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10067, 'log_dups_size': 0, 'ondisk_log_size': 10067, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1586444, 'num_objects': 194, 'num_object_clones': 0, 'num_object_copies': 776, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 194, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12565, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2280}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12281", 'reported_seq': 12352, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.395686+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:01.395686+0000', 'last_peered': '2024-09-15T21:15:01.395686+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:01.395686+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1395224, 'num_objects': 171, 'num_object_clones': 0, 'num_object_copies': 684, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 171, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12278, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2161}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12421", 'reported_seq': 12513, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.765398+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:01.765398+0000', 'last_peered': '2024-09-15T21:15:01.765398+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:01.765398+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10021, 'log_dups_size': 0, 'ondisk_log_size': 10021, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1152268, 'num_objects': 141, 'num_object_clones': 0, 'num_object_copies': 564, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 141, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12419, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2194}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12282", 'reported_seq': 12332, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.377055+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:01.377055+0000', 'last_peered': '2024-09-15T21:15:01.377055+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:01.377055+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10082, 'log_dups_size': 0, 'ondisk_log_size': 10082, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1186438, 'num_objects': 145, 'num_object_clones': 0, 'num_object_copies': 580, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 145, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12281, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2111}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12279", 'reported_seq': 12347, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.486734+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:01.486734+0000', 'last_peered': '2024-09-15T21:15:01.486734+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:01.486734+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1250572, 'num_objects': 153, 'num_object_clones': 0, 'num_object_copies': 612, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 153, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12277, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2126}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:03.968 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12515", 'reported_seq': 12609, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:01.831618+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:01.831618+0000', 'last_peered': '2024-09-15T21:15:01.831618+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:01.831618+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10015, 'log_dups_size': 0, 'ondisk_log_size': 10015, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1357068, 'num_objects': 166, 'num_object_clones': 0, 'num_object_copies': 664, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 166, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12513, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2115}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:03.969 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:15:03.969 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12359", 'reported_seq': 13310, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:14:58.818041+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:14:58.818041+0000', 'last_peered': '2024-09-15T21:14:58.818041+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:14:58.818041+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1553676, 'num_objects': 190, 'num_object_clones': 0, 'num_object_copies': 760, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1882, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 190, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12357, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 437, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1692}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:03.969 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:04.733 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:04.733 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29684/9536 objects degraded (311.284%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":770,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":2384,"data_bytes":19490716,"bytes_used":2722230272,"bytes_avail":383824826368,"bytes_total":386547056640,"degraded_objects":29684,"degraded_total":9536,"degraded_ratio":3.1128355704697985,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":184},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:04.760 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:04.761 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:05.406 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:05.407 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:05.428 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-15T21:15:04.207562+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19490716,"num_objects":2384,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170956,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160838,"ondisk_log_size":160838,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2658428,"kb_used_data":41640,"kb_used_omap":14,"kb_used_meta":2616689,"kb_avail":374828932,"statfs":{"total":386547056640,"available":383824826368,"internally_reserved":0,"allocated":42639360,"data_stored":41914204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2679490098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,4,3,21,16,13,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18153472,"num_objects":-2216,"num_object_clones":0,"num_object_copies":-8864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004988"},"pg_stats":[{"pgid":"1.f","version":"37'12199","reported_seq":12256,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.390170+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:01.390170+0000","last_peered":"2024-09-15T21:15:01.390170+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:01.390170+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1253376,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12199,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12380","reported_seq":17278,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.357235+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:00.357235+0000","last_peered":"2024-09-15T21:15:00.357235+0000","last_clean":"2024-09-15T21:15:00.357235+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:00.357235+0000","last_undegraded":"2024-09-15T21:15:00.357235+0000","last_fullsized":"2024-09-15T21:15:00.357235+0000","mapping_epoch":34,"log_start":"27'2300","ondisk_log_start":"27'2300","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1392640,"num_objects":170,"num_object_clones":0,"num_object_copies":680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3046,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12270","reported_seq":12361,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.611021+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:01.611021+0000","last_peered":"2024-09-15T21:15:01.611021+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:01.611021+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1325726,"num_objects":163,"num_object_clones":0,"num_object_copies":652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12269,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2095}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12320","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.593284+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:03.593284+0000","last_peered":"2024-09-15T21:15:03.593284+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:03.593284+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111308,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12520","reported_seq":12587,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.451175+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:01.451175+0000","last_peered":"2024-09-15T21:15:01.451175+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:01.451175+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1135884,"num_objects":139,"num_object_clones":0,"num_object_copies":556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12518,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2172}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12227","reported_seq":12283,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.400983+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:01.400983+0000","last_peered":"2024-09-15T21:15:01.400983+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:01.400983+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1220608,"num_objects":149,"num_object_clones":0,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12227,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2167}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12660","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.482237+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:03.482237+0000","last_peered":"2024-09-15T21:15:03.482237+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:03.482237+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":914700,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2254}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12626","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.941490+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:03.941490+0000","last_peered":"2024-09-15T21:15:03.941490+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:03.941490+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12624,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12281","reported_seq":12352,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.395686+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:01.395686+0000","last_peered":"2024-09-15T21:15:01.395686+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:01.395686+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1395224,"num_objects":171,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12278,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2161}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12421","reported_seq":12513,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.765398+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:01.765398+0000","last_peered":"2024-09-15T21:15:01.765398+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:01.765398+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1152268,"num_objects":141,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12419,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2194}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12196","reported_seq":16870,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:01.492382+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:01.492382+0000","last_peered":"2024-09-15T21:15:01.492382+0000","last_clean":"2024-09-15T21:15:01.492382+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:01.492382+0000","last_undegraded":"2024-09-15T21:15:01.492382+0000","last_fullsized":"2024-09-15T21:15:01.492382+0000","mapping_epoch":32,"log_start":"27'2100","ondisk_log_start":"27'2100","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1343488,"num_objects":164,"num_object_clones":0,"num_object_copies":656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6307,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12282","reported_seq":12332,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.377055+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:01.377055+0000","last_peered":"2024-09-15T21:15:01.377055+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:01.377055+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1186438,"num_objects":145,"num_object_clones":0,"num_object_copies":580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2111}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12334","reported_seq":17028,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:00.178190+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:00.178190+0000","last_peered":"2024-09-15T21:15:00.178190+0000","last_clean":"2024-09-15T21:15:00.178190+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:00.178190+0000","last_undegraded":"2024-09-15T21:15:00.178190+0000","last_fullsized":"2024-09-15T21:15:00.178190+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1359872,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":596,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12279","reported_seq":12347,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.486734+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:01.486734+0000","last_peered":"2024-09-15T21:15:01.486734+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:01.486734+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1250572,"num_objects":153,"num_object_clones":0,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12277,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2126}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12515","reported_seq":12609,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:01.831618+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:01.831618+0000","last_peered":"2024-09-15T21:15:01.831618+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:01.831618+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1357068,"num_objects":166,"num_object_clones":0,"num_object_copies":664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2115}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12428","reported_seq":13449,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.909489+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:03.909489+0000","last_peered":"2024-09-15T21:15:03.909489+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:03.909489+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988428,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":19490716,"num_objects":2384,"num_object_clones":0,"num_object_copies":9536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2384,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170956,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39481344,"data_stored":39481344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160838,"ondisk_log_size":160838,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":687612,"kb_used_data":10344,"kb_used_omap":4,"kb_used_meta":677243,"kb_avail":93684228,"statfs":{"total":96636764160,"available":95932649472,"internally_reserved":0,"allocated":10592256,"data_stored":10410967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":693497552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,8,10,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691668,"kb_used_data":10756,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93680172,"statfs":{"total":96636764160,"available":95928496128,"internally_reserved":0,"allocated":11014144,"data_stored":10832855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,0,2,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051716,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":566852,"kb_used_data":11812,"kb_used_omap":4,"kb_used_meta":555003,"kb_avail":93804988,"statfs":{"total":96636764160,"available":96056307712,"internally_reserved":0,"allocated":12095488,"data_stored":11914199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":568323791},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,0,4,4,8,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712296,"kb_used_data":8728,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93659544,"statfs":{"total":96636764160,"available":95907373056,"internally_reserved":0,"allocated":8937472,"data_stored":8756183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":8282112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11177984,"data_stored":11177984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9842688,"data_stored":9842688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10178560,"data_stored":10178560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:08.431 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:09.026 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:09.027 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:09.045 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-15T21:15:08.208685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13641628,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171670,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659472,"kb_used_data":30196,"kb_used_omap":14,"kb_used_meta":2629169,"kb_avail":374827888,"statfs":{"total":386547056640,"available":383823757312,"internally_reserved":0,"allocated":30920704,"data_stored":30195548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2692269618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,15,9,16,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20881408,"num_objects":-2549,"num_object_clones":0,"num_object_copies":-10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004373"},"pg_stats":[{"pgid":"1.f","version":"37'12261","reported_seq":12318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.075982+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:06.075982+0000","last_peered":"2024-09-15T21:15:06.075982+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:06.075982+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":745472,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12437","reported_seq":17335,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.007216+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:05.007216+0000","last_peered":"2024-09-15T21:15:05.007216+0000","last_clean":"2024-09-15T21:15:05.007216+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:05.007216+0000","last_undegraded":"2024-09-15T21:15:05.007216+0000","last_fullsized":"2024-09-15T21:15:05.007216+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925696,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12329","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.933545+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:06.933545+0000","last_peered":"2024-09-15T21:15:06.933545+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:06.933545+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12328,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12320","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.593284+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:03.593284+0000","last_peered":"2024-09-15T21:15:03.593284+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:03.593284+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111308,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12575","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.510396+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:06.510396+0000","last_peered":"2024-09-15T21:15:06.510396+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:06.510396+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":685324,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12573,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12281","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.375621+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:06.375621+0000","last_peered":"2024-09-15T21:15:06.375621+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:06.375621+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":778240,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12660","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.482237+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:03.482237+0000","last_peered":"2024-09-15T21:15:03.482237+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:03.482237+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":914700,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2254}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12626","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.941490+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:03.941490+0000","last_peered":"2024-09-15T21:15:03.941490+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:03.941490+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12624,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12354","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.465115+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:06.465115+0000","last_peered":"2024-09-15T21:15:06.465115+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:06.465115+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":797208,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12479","reported_seq":12571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.961587+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:06.961587+0000","last_peered":"2024-09-15T21:15:06.961587+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:06.961587+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":677132,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12264","reported_seq":16938,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:06.362994+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:06.362994+0000","last_peered":"2024-09-15T21:15:06.362994+0000","last_clean":"2024-09-15T21:15:06.362994+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:06.362994+0000","last_undegraded":"2024-09-15T21:15:06.362994+0000","last_fullsized":"2024-09-15T21:15:06.362994+0000","mapping_epoch":32,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.313590+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:06.313590+0000","last_peered":"2024-09-15T21:15:06.313590+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:06.313590+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":793222,"num_objects":97,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12382","reported_seq":17076,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.408397+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:05.408397+0000","last_peered":"2024-09-15T21:15:05.408397+0000","last_clean":"2024-09-15T21:15:05.408397+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:05.408397+0000","last_undegraded":"2024-09-15T21:15:05.408397+0000","last_fullsized":"2024-09-15T21:15:05.408397+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966656,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12339","reported_seq":12407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.352106+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:06.352106+0000","last_peered":"2024-09-15T21:15:06.352106+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:06.352106+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759052,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12337,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12587","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.729950+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:06.729950+0000","last_peered":"2024-09-15T21:15:06.729950+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:06.729950+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":767244,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12428","reported_seq":13449,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.909489+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:03.909489+0000","last_peered":"2024-09-15T21:15:03.909489+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:03.909489+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988428,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13641628,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171670,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27824128,"data_stored":27824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688064,"kb_used_data":6640,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93683776,"statfs":{"total":96636764160,"available":95932186624,"internally_reserved":0,"allocated":6799360,"data_stored":6618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691992,"kb_used_data":6892,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93679848,"statfs":{"total":96636764160,"available":95928164352,"internally_reserved":0,"allocated":7057408,"data_stored":6876119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,2,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567120,"kb_used_data":7936,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93804720,"statfs":{"total":96636764160,"available":96056033280,"internally_reserved":0,"allocated":8126464,"data_stored":7945175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":572583631},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712296,"kb_used_data":8728,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93659544,"statfs":{"total":96636764160,"available":95907373056,"internally_reserved":0,"allocated":8937472,"data_stored":8756183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":8282112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7254016,"data_stored":7254016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6340608,"data_stored":6340608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:09.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:09.654 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:09.654 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:09.671 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":415,"stamp":"2024-09-15T21:15:08.208685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13641628,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171670,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659472,"kb_used_data":30196,"kb_used_omap":14,"kb_used_meta":2629169,"kb_avail":374827888,"statfs":{"total":386547056640,"available":383823757312,"internally_reserved":0,"allocated":30920704,"data_stored":30195548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2692269618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,15,9,16,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20881408,"num_objects":-2549,"num_object_clones":0,"num_object_copies":-10196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2549,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004373"},"pg_stats":[{"pgid":"1.f","version":"37'12261","reported_seq":12318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.075982+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:06.075982+0000","last_peered":"2024-09-15T21:15:06.075982+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:06.075982+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":745472,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12437","reported_seq":17335,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.007216+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:05.007216+0000","last_peered":"2024-09-15T21:15:05.007216+0000","last_clean":"2024-09-15T21:15:05.007216+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:05.007216+0000","last_undegraded":"2024-09-15T21:15:05.007216+0000","last_fullsized":"2024-09-15T21:15:05.007216+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925696,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12329","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.933545+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:06.933545+0000","last_peered":"2024-09-15T21:15:06.933545+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:06.933545+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12328,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12320","reported_seq":12400,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.593284+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:03.593284+0000","last_peered":"2024-09-15T21:15:03.593284+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:03.593284+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1111308,"num_objects":136,"num_object_clones":0,"num_object_copies":544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12575","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.510396+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:06.510396+0000","last_peered":"2024-09-15T21:15:06.510396+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:06.510396+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":685324,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12573,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12281","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.375621+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:06.375621+0000","last_peered":"2024-09-15T21:15:06.375621+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:06.375621+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":778240,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12660","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.482237+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:03.482237+0000","last_peered":"2024-09-15T21:15:03.482237+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:03.482237+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":914700,"num_objects":112,"num_object_clones":0,"num_object_copies":448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12658,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2254}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12626","reported_seq":12705,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.941490+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:03.941490+0000","last_peered":"2024-09-15T21:15:03.941490+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:03.941490+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1103116,"num_objects":135,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12624,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2339}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12354","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.465115+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:06.465115+0000","last_peered":"2024-09-15T21:15:06.465115+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:06.465115+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":797208,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12479","reported_seq":12571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.961587+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:06.961587+0000","last_peered":"2024-09-15T21:15:06.961587+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:06.961587+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":677132,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12264","reported_seq":16938,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:06.362994+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:06.362994+0000","last_peered":"2024-09-15T21:15:06.362994+0000","last_clean":"2024-09-15T21:15:06.362994+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:06.362994+0000","last_undegraded":"2024-09-15T21:15:06.362994+0000","last_fullsized":"2024-09-15T21:15:06.362994+0000","mapping_epoch":32,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.313590+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:06.313590+0000","last_peered":"2024-09-15T21:15:06.313590+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:06.313590+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":793222,"num_objects":97,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12382","reported_seq":17076,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.408397+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:05.408397+0000","last_peered":"2024-09-15T21:15:05.408397+0000","last_clean":"2024-09-15T21:15:05.408397+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:05.408397+0000","last_undegraded":"2024-09-15T21:15:05.408397+0000","last_fullsized":"2024-09-15T21:15:05.408397+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966656,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12339","reported_seq":12407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.352106+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:06.352106+0000","last_peered":"2024-09-15T21:15:06.352106+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:06.352106+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759052,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12337,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12587","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.729950+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:06.729950+0000","last_peered":"2024-09-15T21:15:06.729950+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:06.729950+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":767244,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12428","reported_seq":13449,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:03.909489+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:03.909489+0000","last_peered":"2024-09-15T21:15:03.909489+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:03.909489+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":988428,"num_objects":121,"num_object_clones":0,"num_object_copies":484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":472,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1726}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":13641628,"num_objects":1670,"num_object_clones":0,"num_object_copies":6680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1670,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171670,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":491,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":27824128,"data_stored":27824128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160852,"ondisk_log_size":160852,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688064,"kb_used_data":6640,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93683776,"statfs":{"total":96636764160,"available":95932186624,"internally_reserved":0,"allocated":6799360,"data_stored":6618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691992,"kb_used_data":6892,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93679848,"statfs":{"total":96636764160,"available":95928164352,"internally_reserved":0,"allocated":7057408,"data_stored":6876119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,2,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567120,"kb_used_data":7936,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93804720,"statfs":{"total":96636764160,"available":96056033280,"internally_reserved":0,"allocated":8126464,"data_stored":7945175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":572583631},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247955,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712296,"kb_used_data":8728,"kb_used_omap":4,"kb_used_meta":703547,"kb_avail":93659544,"statfs":{"total":96636764160,"available":95907373056,"internally_reserved":0,"allocated":8937472,"data_stored":8756183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":720432846},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8282112,"data_stored":8282112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7254016,"data_stored":7254016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6340608,"data_stored":6340608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:09.672 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12261", 'reported_seq': 12318, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.075982+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:15:06.075982+0000', 'last_peered': '2024-09-15T21:15:06.075982+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:06.075982+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10061, 'log_dups_size': 0, 'ondisk_log_size': 10061, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 745472, 'num_objects': 91, 'num_object_clones': 0, 'num_object_copies': 364, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 91, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12261, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2185}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12329", 'reported_seq': 12420, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.933545+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:06.933545+0000', 'last_peered': '2024-09-15T21:15:06.933545+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:06.933545+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10029, 'log_dups_size': 0, 'ondisk_log_size': 10029, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 842398, 'num_objects': 104, 'num_object_clones': 0, 'num_object_copies': 416, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 104, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12328, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2154}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12320", 'reported_seq': 12400, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:03.593284+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:03.593284+0000', 'last_peered': '2024-09-15T21:15:03.593284+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:03.593284+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10020, 'log_dups_size': 0, 'ondisk_log_size': 10020, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1111308, 'num_objects': 136, 'num_object_clones': 0, 'num_object_copies': 544, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 136, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12318, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12575", 'reported_seq': 12641, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.510396+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:06.510396+0000', 'last_peered': '2024-09-15T21:15:06.510396+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:06.510396+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10075, 'log_dups_size': 0, 'ondisk_log_size': 10075, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 685324, 'num_objects': 84, 'num_object_clones': 0, 'num_object_copies': 336, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 84, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12573, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2227}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:09.673 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12281", 'reported_seq': 12337, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.375621+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:06.375621+0000', 'last_peered': '2024-09-15T21:15:06.375621+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:06.375621+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "27'2200", 'ondisk_log_start': "27'2200", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10081, 'log_dups_size': 0, 'ondisk_log_size': 10081, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 778240, 'num_objects': 95, 'num_object_clones': 0, 'num_object_copies': 380, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 95, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12281, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2221}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12660", 'reported_seq': 12741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:03.482237+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:03.482237+0000', 'last_peered': '2024-09-15T21:15:03.482237+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:03.482237+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 914700, 'num_objects': 112, 'num_object_clones': 0, 'num_object_copies': 448, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 112, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12658, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2254}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12626", 'reported_seq': 12705, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:03.941490+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:03.941490+0000', 'last_peered': '2024-09-15T21:15:03.941490+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:03.941490+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1103116, 'num_objects': 135, 'num_object_clones': 0, 'num_object_copies': 540, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 135, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12624, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2339}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12354", 'reported_seq': 12425, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.465115+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:06.465115+0000', 'last_peered': '2024-09-15T21:15:06.465115+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:06.465115+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 797208, 'num_objects': 98, 'num_object_clones': 0, 'num_object_copies': 392, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 98, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12351, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2234}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12479", 'reported_seq': 12571, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.961587+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:06.961587+0000', 'last_peered': '2024-09-15T21:15:06.961587+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:06.961587+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 677132, 'num_objects': 83, 'num_object_clones': 0, 'num_object_copies': 332, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 83, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12477, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2252}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12330", 'reported_seq': 12381, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.313590+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:06.313590+0000', 'last_peered': '2024-09-15T21:15:06.313590+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:06.313590+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 793222, 'num_objects': 97, 'num_object_clones': 0, 'num_object_copies': 388, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 97, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12329, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2159}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.674 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:09.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12339", 'reported_seq': 12407, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.352106+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:06.352106+0000', 'last_peered': '2024-09-15T21:15:06.352106+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:06.352106+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10039, 'log_dups_size': 0, 'ondisk_log_size': 10039, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 759052, 'num_objects': 93, 'num_object_clones': 0, 'num_object_copies': 372, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 93, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12337, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2186}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:09.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:09.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12587", 'reported_seq': 12681, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:06.729950+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:06.729950+0000', 'last_peered': '2024-09-15T21:15:06.729950+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:06.729950+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10087, 'log_dups_size': 0, 'ondisk_log_size': 10087, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 767244, 'num_objects': 94, 'num_object_clones': 0, 'num_object_copies': 376, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 94, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12585, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2187}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:09.675 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:15:09.675 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12428", 'reported_seq': 13449, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:03.909489+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:15:03.909489+0000', 'last_peered': '2024-09-15T21:15:03.909489+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:15:03.909489+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10028, 'log_dups_size': 0, 'ondisk_log_size': 10028, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 988428, 'num_objects': 121, 'num_object_clones': 0, 'num_object_copies': 484, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1847, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 121, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12426, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 472, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1726}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:09.675 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:10.467 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:10.467 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29684/6680 objects degraded (444.371%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":776,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":1670,"data_bytes":13641628,"bytes_used":2723299328,"bytes_avail":383823757312,"bytes_total":386547056640,"degraded_objects":29684,"degraded_total":6680,"degraded_ratio":4.4437125748502995,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":75,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:10.495 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:10.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:11.131 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:11.132 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:11.149 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-15T21:15:10.209386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":11683740,"num_objects":1431,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171909,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2659756,"kb_used_data":26320,"kb_used_omap":14,"kb_used_meta":2633329,"kb_avail":374827604,"statfs":{"total":386547056640,"available":383823466496,"internally_reserved":0,"allocated":26951680,"data_stored":26226524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2696529458},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,1,8,10,18,13,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17874944,"num_objects":-2182,"num_object_clones":0,"num_object_copies":-8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004209"},"pg_stats":[{"pgid":"1.f","version":"37'12261","reported_seq":12318,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.075982+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:06.075982+0000","last_peered":"2024-09-15T21:15:06.075982+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:06.075982+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":745472,"num_objects":91,"num_object_clones":0,"num_object_copies":364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12261,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12437","reported_seq":17335,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.007216+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:05.007216+0000","last_peered":"2024-09-15T21:15:05.007216+0000","last_clean":"2024-09-15T21:15:05.007216+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:05.007216+0000","last_undegraded":"2024-09-15T21:15:05.007216+0000","last_fullsized":"2024-09-15T21:15:05.007216+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":925696,"num_objects":113,"num_object_clones":0,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3103,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12329","reported_seq":12420,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.933545+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:06.933545+0000","last_peered":"2024-09-15T21:15:06.933545+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:06.933545+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":842398,"num_objects":104,"num_object_clones":0,"num_object_copies":416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12328,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2154}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12379","reported_seq":12459,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:08.994850+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:08.994850+0000","last_peered":"2024-09-15T21:15:08.994850+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:08.994850+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":627980,"num_objects":77,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12377,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2257}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12575","reported_seq":12641,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.510396+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:06.510396+0000","last_peered":"2024-09-15T21:15:06.510396+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:06.510396+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":685324,"num_objects":84,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12573,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12281","reported_seq":12337,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.375621+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:06.375621+0000","last_peered":"2024-09-15T21:15:06.375621+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:06.375621+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":778240,"num_objects":95,"num_object_clones":0,"num_object_copies":380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12281,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2221}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12714","reported_seq":12795,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:08.922327+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:08.922327+0000","last_peered":"2024-09-15T21:15:08.922327+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:08.922327+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472332,"num_objects":58,"num_object_clones":0,"num_object_copies":232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12712,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12693","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:08.914708+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:08.914708+0000","last_peered":"2024-09-15T21:15:08.914708+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:08.914708+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554252,"num_objects":68,"num_object_clones":0,"num_object_copies":272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12691,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2406}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12354","reported_seq":12425,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.465115+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:06.465115+0000","last_peered":"2024-09-15T21:15:06.465115+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:06.465115+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":797208,"num_objects":98,"num_object_clones":0,"num_object_copies":392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12351,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2234}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12479","reported_seq":12571,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.961587+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:06.961587+0000","last_peered":"2024-09-15T21:15:06.961587+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:06.961587+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":677132,"num_objects":83,"num_object_clones":0,"num_object_copies":332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2252}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12264","reported_seq":16938,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:06.362994+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:06.362994+0000","last_peered":"2024-09-15T21:15:06.362994+0000","last_clean":"2024-09-15T21:15:06.362994+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:06.362994+0000","last_undegraded":"2024-09-15T21:15:06.362994+0000","last_fullsized":"2024-09-15T21:15:06.362994+0000","mapping_epoch":32,"log_start":"27'2200","ondisk_log_start":"27'2200","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":96,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6375,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12330","reported_seq":12381,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.313590+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:06.313590+0000","last_peered":"2024-09-15T21:15:06.313590+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:06.313590+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":793222,"num_objects":97,"num_object_clones":0,"num_object_copies":388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12329,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2159}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12382","reported_seq":17076,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:05.408397+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:05.408397+0000","last_peered":"2024-09-15T21:15:05.408397+0000","last_clean":"2024-09-15T21:15:05.408397+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:05.408397+0000","last_undegraded":"2024-09-15T21:15:05.408397+0000","last_fullsized":"2024-09-15T21:15:05.408397+0000","mapping_epoch":36,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":966656,"num_objects":118,"num_object_clones":0,"num_object_copies":472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12339","reported_seq":12407,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.352106+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:06.352106+0000","last_peered":"2024-09-15T21:15:06.352106+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:06.352106+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":759052,"num_objects":93,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12337,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2186}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12587","reported_seq":12681,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:06.729950+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:06.729950+0000","last_peered":"2024-09-15T21:15:06.729950+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:06.729950+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":767244,"num_objects":94,"num_object_clones":0,"num_object_copies":376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":94,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12585,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12487","reported_seq":13586,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:08.934822+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:08.934822+0000","last_peered":"2024-09-15T21:15:08.934822+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:08.934822+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505100,"num_objects":62,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1808,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12485,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1746}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":11683740,"num_objects":1431,"num_object_clones":0,"num_object_copies":5724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171909,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23846912,"data_stored":23846912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160991,"ondisk_log_size":160991,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688064,"kb_used_data":6640,"kb_used_omap":4,"kb_used_meta":681403,"kb_avail":93683776,"statfs":{"total":96636764160,"available":95932186624,"internally_reserved":0,"allocated":6799360,"data_stored":6618071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":697757392},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3,11,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691992,"kb_used_data":6892,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93679848,"statfs":{"total":96636764160,"available":95928164352,"internally_reserved":0,"allocated":7057408,"data_stored":6876119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,2,3,4,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051717,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":567120,"kb_used_data":7936,"kb_used_omap":4,"kb_used_meta":559163,"kb_avail":93804720,"statfs":{"total":96636764160,"available":96056033280,"internally_reserved":0,"allocated":8126464,"data_stored":7945175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":572583631},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247956,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":712580,"kb_used_data":4852,"kb_used_omap":4,"kb_used_meta":707707,"kb_avail":93659260,"statfs":{"total":96636764160,"available":95907082240,"internally_reserved":0,"allocated":4968448,"data_stored":4787159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":724692686},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4304896,"data_stored":4304896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7254016,"data_stored":7254016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5947392,"data_stored":5947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":6340608,"data_stored":6340608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:14.151 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:14.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Removed 100108 objects 2024-09-15T21:15:14.373 INFO:tasks.ec_lost_unfound.radosbench.client.admin.smithi181.stdout:Clean up completed and total clean up time :469.689 2024-09-15T21:15:14.608 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:14.609 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:14.624 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-15T21:15:14.210830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661760,"kb_used_data":10300,"kb_used_omap":14,"kb_used_meta":2651377,"kb_avail":374825600,"statfs":{"total":386547056640,"available":383821414400,"internally_reserved":0,"allocated":10547200,"data_stored":9822044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2715010610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,7,11,11,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-2162,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"1.f","version":"37'12318","reported_seq":12375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.410954+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:11.410954+0000","last_peered":"2024-09-15T21:15:11.410954+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:11.410954+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12495","reported_seq":17393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.378320+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:10.378320+0000","last_peered":"2024-09-15T21:15:10.378320+0000","last_clean":"2024-09-15T21:15:10.378320+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:10.378320+0000","last_undegraded":"2024-09-15T21:15:10.378320+0000","last_fullsized":"2024-09-15T21:15:10.378320+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12402","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.981329+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:11.981329+0000","last_peered":"2024-09-15T21:15:11.981329+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:11.981329+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244382,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12401,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12634","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.449291+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:11.449291+0000","last_peered":"2024-09-15T21:15:11.449291+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:11.449291+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201996,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12336","reported_seq":12392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.463553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:11.463553+0000","last_peered":"2024-09-15T21:15:11.463553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:11.463553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":12482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.506579+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:11.506579+0000","last_peered":"2024-09-15T21:15:11.506579+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:11.506579+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330264,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12534","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.870026+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:11.870026+0000","last_peered":"2024-09-15T21:15:11.870026+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:11.870026+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226572,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12320","reported_seq":16993,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:11.509424+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:11.509424+0000","last_peered":"2024-09-15T21:15:11.509424+0000","last_clean":"2024-09-15T21:15:11.509424+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:11.509424+0000","last_undegraded":"2024-09-15T21:15:11.509424+0000","last_fullsized":"2024-09-15T21:15:11.509424+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12391","reported_seq":12442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.455969+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:11.455969+0000","last_peered":"2024-09-15T21:15:11.455969+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:11.455969+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293510,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12449","reported_seq":17143,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.525092+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:10.525092+0000","last_peered":"2024-09-15T21:15:10.525092+0000","last_clean":"2024-09-15T21:15:10.525092+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:10.525092+0000","last_undegraded":"2024-09-15T21:15:10.525092+0000","last_fullsized":"2024-09-15T21:15:10.525092+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12401","reported_seq":12469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.561916+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:11.561916+0000","last_peered":"2024-09-15T21:15:11.561916+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:11.561916+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251148,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.932373+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:11.932373+0000","last_peered":"2024-09-15T21:15:11.932373+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:11.932373+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275724,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7761920,"data_stored":7761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688400,"kb_used_data":2812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683440,"statfs":{"total":96636764160,"available":95931842560,"internally_reserved":0,"allocated":2879488,"data_stored":2698199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691592,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93680248,"statfs":{"total":96636764160,"available":95928573952,"internally_reserved":0,"allocated":2416640,"data_stored":2235351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585448,"kb_used_data":4056,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93786392,"statfs":{"total":96636764160,"available":96037265408,"internally_reserved":0,"allocated":4153344,"data_stored":3972055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3244032,"data_stored":3244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1974272,"data_stored":1974272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:14.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:14.932 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:14.932 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:14.945 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-15T21:15:14.210830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661760,"kb_used_data":10300,"kb_used_omap":14,"kb_used_meta":2651377,"kb_avail":374825600,"statfs":{"total":386547056640,"available":383821414400,"internally_reserved":0,"allocated":10547200,"data_stored":9822044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2715010610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,7,11,11,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-2162,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"1.f","version":"37'12318","reported_seq":12375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.410954+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:11.410954+0000","last_peered":"2024-09-15T21:15:11.410954+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:11.410954+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12495","reported_seq":17393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.378320+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:10.378320+0000","last_peered":"2024-09-15T21:15:10.378320+0000","last_clean":"2024-09-15T21:15:10.378320+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:10.378320+0000","last_undegraded":"2024-09-15T21:15:10.378320+0000","last_fullsized":"2024-09-15T21:15:10.378320+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12402","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.981329+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:11.981329+0000","last_peered":"2024-09-15T21:15:11.981329+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:11.981329+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244382,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12401,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12634","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.449291+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:11.449291+0000","last_peered":"2024-09-15T21:15:11.449291+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:11.449291+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201996,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12336","reported_seq":12392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.463553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:11.463553+0000","last_peered":"2024-09-15T21:15:11.463553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:11.463553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":12482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.506579+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:11.506579+0000","last_peered":"2024-09-15T21:15:11.506579+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:11.506579+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330264,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12534","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.870026+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:11.870026+0000","last_peered":"2024-09-15T21:15:11.870026+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:11.870026+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226572,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12320","reported_seq":16993,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:11.509424+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:11.509424+0000","last_peered":"2024-09-15T21:15:11.509424+0000","last_clean":"2024-09-15T21:15:11.509424+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:11.509424+0000","last_undegraded":"2024-09-15T21:15:11.509424+0000","last_fullsized":"2024-09-15T21:15:11.509424+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12391","reported_seq":12442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.455969+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:11.455969+0000","last_peered":"2024-09-15T21:15:11.455969+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:11.455969+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293510,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12449","reported_seq":17143,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.525092+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:10.525092+0000","last_peered":"2024-09-15T21:15:10.525092+0000","last_clean":"2024-09-15T21:15:10.525092+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:10.525092+0000","last_undegraded":"2024-09-15T21:15:10.525092+0000","last_fullsized":"2024-09-15T21:15:10.525092+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12401","reported_seq":12469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.561916+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:11.561916+0000","last_peered":"2024-09-15T21:15:11.561916+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:11.561916+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251148,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.932373+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:11.932373+0000","last_peered":"2024-09-15T21:15:11.932373+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:11.932373+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275724,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7761920,"data_stored":7761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688400,"kb_used_data":2812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683440,"statfs":{"total":96636764160,"available":95931842560,"internally_reserved":0,"allocated":2879488,"data_stored":2698199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691592,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93680248,"statfs":{"total":96636764160,"available":95928573952,"internally_reserved":0,"allocated":2416640,"data_stored":2235351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585448,"kb_used_data":4056,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93786392,"statfs":{"total":96636764160,"available":96037265408,"internally_reserved":0,"allocated":4153344,"data_stored":3972055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3244032,"data_stored":3244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1974272,"data_stored":1974272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:14.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:14.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12318", 'reported_seq': 12375, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.410954+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:15:11.410954+0000', 'last_peered': '2024-09-15T21:15:11.410954+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:11.410954+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10018, 'log_dups_size': 0, 'ondisk_log_size': 10018, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 278528, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 34, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12318, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2242}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.946 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:14.946 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12402", 'reported_seq': 12493, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.981329+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:11.981329+0000', 'last_peered': '2024-09-15T21:15:11.981329+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:11.981329+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10002, 'log_dups_size': 0, 'ondisk_log_size': 10002, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 244382, 'num_objects': 31, 'num_object_clones': 0, 'num_object_copies': 124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 31, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12401, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2227}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12451", 'reported_seq': 12532, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.117856+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:14.117856+0000', 'last_peered': '2024-09-15T21:15:14.117856+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:14.117856+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38156, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12449, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12634", 'reported_seq': 12701, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.449291+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:11.449291+0000', 'last_peered': '2024-09-15T21:15:11.449291+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:11.449291+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 201996, 'num_objects': 25, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 25, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12632, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2286}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12336", 'reported_seq': 12392, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.463553+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:11.463553+0000', 'last_peered': '2024-09-15T21:15:11.463553+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:11.463553+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10036, 'log_dups_size': 0, 'ondisk_log_size': 10036, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 327680, 'num_objects': 40, 'num_object_clones': 0, 'num_object_copies': 160, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 40, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12336, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12766", 'reported_seq': 12847, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:13.810584+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:13.810584+0000', 'last_peered': '2024-09-15T21:15:13.810584+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:13.810584+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46348, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12764, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2360}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:14.947 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12754", 'reported_seq': 12834, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:13.958639+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:13.958639+0000', 'last_peered': '2024-09-15T21:15:13.958639+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:13.958639+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 54540, 'num_objects': 7, 'num_object_clones': 0, 'num_object_copies': 28, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12752, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12411", 'reported_seq': 12482, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.506579+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:11.506579+0000', 'last_peered': '2024-09-15T21:15:11.506579+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:11.506579+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10011, 'log_dups_size': 0, 'ondisk_log_size': 10011, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 330264, 'num_objects': 41, 'num_object_clones': 0, 'num_object_copies': 164, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 41, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12408, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2291}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12534", 'reported_seq': 12626, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.870026+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:11.870026+0000', 'last_peered': '2024-09-15T21:15:11.870026+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:11.870026+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10034, 'log_dups_size': 0, 'ondisk_log_size': 10034, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 226572, 'num_objects': 28, 'num_object_clones': 0, 'num_object_copies': 112, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 28, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12532, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2307}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12391", 'reported_seq': 12442, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.455969+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:11.455969+0000', 'last_peered': '2024-09-15T21:15:11.455969+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:11.455969+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10091, 'log_dups_size': 0, 'ondisk_log_size': 10091, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 293510, 'num_objects': 36, 'num_object_clones': 0, 'num_object_copies': 144, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 36, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12390, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2220}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12401", 'reported_seq': 12469, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.561916+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:11.561916+0000', 'last_peered': '2024-09-15T21:15:11.561916+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:11.561916+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10001, 'log_dups_size': 0, 'ondisk_log_size': 10001, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 251148, 'num_objects': 31, 'num_object_clones': 0, 'num_object_copies': 124, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 31, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12399, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2248}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:14.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12647", 'reported_seq': 12741, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:11.932373+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:11.932373+0000', 'last_peered': '2024-09-15T21:15:11.932373+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:11.932373+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 275724, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 136, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 34, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12645, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2247}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:14.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:15:14.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12543", 'reported_seq': 13738, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.142237+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:15:14.142237+0000', 'last_peered': '2024-09-15T21:15:14.142237+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:15:14.142237+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46348, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12541, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 559, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:14.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:15.367 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:15.367 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29597/1880 objects degraded (1574.309%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":781,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":470,"data_bytes":3811228,"bytes_used":2725642240,"bytes_avail":383821414400,"bytes_total":386547056640,"degraded_objects":29597,"degraded_total":1880,"degraded_ratio":15.74308510638298,"recovering_objects_per_sec":10,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":180},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:15.381 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:15.381 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:15.707 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:15.707 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:15.720 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-15T21:15:14.210830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2661760,"kb_used_data":10300,"kb_used_omap":14,"kb_used_meta":2651377,"kb_avail":374825600,"statfs":{"total":386547056640,"available":383821414400,"internally_reserved":0,"allocated":10547200,"data_stored":9822044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2715010610},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,7,11,11,15,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17711104,"num_objects":-2162,"num_object_clones":0,"num_object_copies":-8648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003900"},"pg_stats":[{"pgid":"1.f","version":"37'12318","reported_seq":12375,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.410954+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:11.410954+0000","last_peered":"2024-09-15T21:15:11.410954+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:11.410954+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12318,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12495","reported_seq":17393,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.378320+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:10.378320+0000","last_peered":"2024-09-15T21:15:10.378320+0000","last_clean":"2024-09-15T21:15:10.378320+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:10.378320+0000","last_undegraded":"2024-09-15T21:15:10.378320+0000","last_fullsized":"2024-09-15T21:15:10.378320+0000","mapping_epoch":34,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450560,"num_objects":55,"num_object_clones":0,"num_object_copies":220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12402","reported_seq":12493,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.981329+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:11.981329+0000","last_peered":"2024-09-15T21:15:11.981329+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:11.981329+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244382,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12401,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2227}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12634","reported_seq":12701,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.449291+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:11.449291+0000","last_peered":"2024-09-15T21:15:11.449291+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:11.449291+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201996,"num_objects":25,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2286}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12336","reported_seq":12392,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.463553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:11.463553+0000","last_peered":"2024-09-15T21:15:11.463553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:11.463553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12336,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12411","reported_seq":12482,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.506579+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:11.506579+0000","last_peered":"2024-09-15T21:15:11.506579+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:11.506579+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330264,"num_objects":41,"num_object_clones":0,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12408,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2291}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12534","reported_seq":12626,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.870026+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:11.870026+0000","last_peered":"2024-09-15T21:15:11.870026+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:11.870026+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226572,"num_objects":28,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12532,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2307}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12320","reported_seq":16993,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:11.509424+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:11.509424+0000","last_peered":"2024-09-15T21:15:11.509424+0000","last_clean":"2024-09-15T21:15:11.509424+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:11.509424+0000","last_undegraded":"2024-09-15T21:15:11.509424+0000","last_fullsized":"2024-09-15T21:15:11.509424+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":40,"num_object_clones":0,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6431,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12391","reported_seq":12442,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.455969+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:11.455969+0000","last_peered":"2024-09-15T21:15:11.455969+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:11.455969+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293510,"num_objects":36,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12390,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2220}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12449","reported_seq":17143,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:10.525092+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:10.525092+0000","last_peered":"2024-09-15T21:15:10.525092+0000","last_clean":"2024-09-15T21:15:10.525092+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:10.525092+0000","last_undegraded":"2024-09-15T21:15:10.525092+0000","last_fullsized":"2024-09-15T21:15:10.525092+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792,"num_objects":51,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":711,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12401","reported_seq":12469,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.561916+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:11.561916+0000","last_peered":"2024-09-15T21:15:11.561916+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:11.561916+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251148,"num_objects":31,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12399,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2248}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12647","reported_seq":12741,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:11.932373+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:11.932373+0000","last_peered":"2024-09-15T21:15:11.932373+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:11.932373+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275724,"num_objects":34,"num_object_clones":0,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12645,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2247}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":3811228,"num_objects":470,"num_object_clones":0,"num_object_copies":1880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172870,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7761920,"data_stored":7761920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160652,"ondisk_log_size":160652,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":688400,"kb_used_data":2812,"kb_used_omap":4,"kb_used_meta":685563,"kb_avail":93683440,"statfs":{"total":96636764160,"available":95931842560,"internally_reserved":0,"allocated":2879488,"data_stored":2698199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":702017232},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,3,6,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":691592,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":93680248,"statfs":{"total":96636764160,"available":95928573952,"internally_reserved":0,"allocated":2416640,"data_stored":2235351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,4,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051718,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585448,"kb_used_data":4056,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93786392,"statfs":{"total":96636764160,"available":96037265408,"internally_reserved":0,"allocated":4153344,"data_stored":3972055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3244032,"data_stored":3244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1974272,"data_stored":1974272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:18.721 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:19.039 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:19.040 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:19.052 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-15T21:15:18.211856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685464,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2683313,"kb_avail":374801896,"statfs":{"total":386547056640,"available":383797141504,"internally_reserved":0,"allocated":2121728,"data_stored":1396572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2747713074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,4,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18350104,"num_objects":-2241,"num_object_clones":0,"num_object_copies":-8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003809"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17448,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.270637+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.270637+0000","last_peered":"2024-09-15T21:15:14.270637+0000","last_clean":"2024-09-15T21:15:14.270637+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.270637+0000","last_undegraded":"2024-09-15T21:15:14.270637+0000","last_fullsized":"2024-09-15T21:15:14.270637+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":274432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585936,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":585531,"kb_avail":93785904,"statfs":{"total":96636764160,"available":96036765696,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":599584463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:19.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:19.366 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:19.367 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:19.380 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-15T21:15:18.211856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685464,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2683313,"kb_avail":374801896,"statfs":{"total":386547056640,"available":383797141504,"internally_reserved":0,"allocated":2121728,"data_stored":1396572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2747713074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,4,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18350104,"num_objects":-2241,"num_object_clones":0,"num_object_copies":-8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003809"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17448,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.270637+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.270637+0000","last_peered":"2024-09-15T21:15:14.270637+0000","last_clean":"2024-09-15T21:15:14.270637+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.270637+0000","last_undegraded":"2024-09-15T21:15:14.270637+0000","last_fullsized":"2024-09-15T21:15:14.270637+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":274432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585936,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":585531,"kb_avail":93785904,"statfs":{"total":96636764160,"available":96036765696,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":599584463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:19.380 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12352", 'reported_seq': 12409, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.059005+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:15:14.059005+0000', 'last_peered': '2024-09-15T21:15:14.059005+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:14.059005+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12352, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.243109+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:14.243109+0000', 'last_peered': '2024-09-15T21:15:14.243109+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:14.243109+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12451", 'reported_seq': 12532, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.117856+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:14.117856+0000', 'last_peered': '2024-09-15T21:15:14.117856+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:14.117856+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10051, 'log_dups_size': 0, 'ondisk_log_size': 10051, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 38156, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 20, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12449, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2329}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12724, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.130081+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:14.130081+0000', 'last_peered': '2024-09-15T21:15:14.130081+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:14.130081+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:19.381 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12432, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.346512+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:14.346512+0000', 'last_peered': '2024-09-15T21:15:14.346512+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:14.346512+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12766", 'reported_seq': 12847, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:13.810584+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:13.810584+0000', 'last_peered': '2024-09-15T21:15:13.810584+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:13.810584+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10066, 'log_dups_size': 0, 'ondisk_log_size': 10066, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46348, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12764, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2360}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12754", 'reported_seq': 12834, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:13.958639+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:13.958639+0000', 'last_peered': '2024-09-15T21:15:13.958639+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:13.958639+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 54540, 'num_objects': 7, 'num_object_clones': 0, 'num_object_copies': 28, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 7, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12752, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2467}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12519, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.144794+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:14.144794+0000', 'last_peered': '2024-09-15T21:15:14.144794+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:14.144794+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.340319+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:14.340319+0000', 'last_peered': '2024-09-15T21:15:14.340319+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:14.340319+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:19.382 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12477, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.279504+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:14.279504+0000', 'last_peered': '2024-09-15T21:15:14.279504+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:14.279504+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.202581+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:14.202581+0000', 'last_peered': '2024-09-15T21:15:14.202581+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:14.202581+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.286579+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:14.286579+0000', 'last_peered': '2024-09-15T21:15:14.286579+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:14.286579+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:15:19.383 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12543", 'reported_seq': 13738, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.142237+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:15:14.142237+0000', 'last_peered': '2024-09-15T21:15:14.142237+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:15:14.142237+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10043, 'log_dups_size': 0, 'ondisk_log_size': 10043, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 46348, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1760, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12541, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 559, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1754}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:19.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:19.873 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:19.873 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 29597/152 objects degraded (19471.711%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":785,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":13},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":38,"data_bytes":280452,"bytes_used":2749915136,"bytes_avail":383797141504,"bytes_total":386547056640,"degraded_objects":29597,"degraded_total":152,"degraded_ratio":194.71710526315789,"recovering_objects_per_sec":7,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":186},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:19.886 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:19.886 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:20.211 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:20.212 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:20.224 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-15T21:15:18.211856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2685464,"kb_used_data":2072,"kb_used_omap":14,"kb_used_meta":2683313,"kb_avail":374801896,"statfs":{"total":386547056640,"available":383797141504,"internally_reserved":0,"allocated":2121728,"data_stored":1396572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2747713074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,4,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-18350104,"num_objects":-2241,"num_object_clones":0,"num_object_copies":-8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003809"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17448,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.270637+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.270637+0000","last_peered":"2024-09-15T21:15:14.270637+0000","last_clean":"2024-09-15T21:15:14.270637+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.270637+0000","last_undegraded":"2024-09-15T21:15:14.270637+0000","last_fullsized":"2024-09-15T21:15:14.270637+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12451","reported_seq":12532,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.117856+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.117856+0000","last_peered":"2024-09-15T21:15:14.117856+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.117856+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38156,"num_objects":5,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12449,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2329}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12766","reported_seq":12847,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.810584+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:13.810584+0000","last_peered":"2024-09-15T21:15:13.810584+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:13.810584+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12764,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2360}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12754","reported_seq":12834,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:13.958639+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:13.958639+0000","last_peered":"2024-09-15T21:15:13.958639+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:13.958639+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54540,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2467}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12543","reported_seq":13738,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.142237+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:14.142237+0000","last_peered":"2024-09-15T21:15:14.142237+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:14.142237+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46348,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12541,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":559,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1754}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":280452,"num_objects":38,"num_object_clones":0,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173302,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":578,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":274432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160884,"ondisk_log_size":160884,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051719,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":585936,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":585531,"kb_avail":93785904,"statfs":{"total":96636764160,"available":96036765696,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":599584463},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247957,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":696320,"kb_used_data":1072,"kb_used_omap":4,"kb_used_meta":695227,"kb_avail":93675520,"statfs":{"total":96636764160,"available":95923732480,"internally_reserved":0,"allocated":1097728,"data_stored":916439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":711913166},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":184320,"data_stored":184320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:23.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:23.542 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:23.542 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:23.554 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-15T21:15:22.212652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690268,"kb_used_data":1308,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797092,"statfs":{"total":386547056640,"available":383792222208,"internally_reserved":0,"allocated":1339392,"data_stored":614236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11534360,"num_objects":-1409,"num_object_clones":0,"num_object_copies":-5636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003265"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17449,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.340616+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.340616+0000","last_peered":"2024-09-15T21:15:14.340616+0000","last_clean":"2024-09-15T21:15:14.340616+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.340616+0000","last_undegraded":"2024-09-15T21:15:14.340616+0000","last_fullsized":"2024-09-15T21:15:14.340616+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.350312+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.350312+0000","last_peered":"2024-09-15T21:15:14.350312+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.350312+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.204352+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:14.204352+0000","last_peered":"2024-09-15T21:15:14.204352+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:14.204352+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12839,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.365155+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:14.365155+0000","last_peered":"2024-09-15T21:15:14.365155+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:14.365155+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":15452,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:19.166010+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:19.166010+0000","last_peered":"2024-09-15T21:15:19.166010+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:19.166010+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573456,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798384,"statfs":{"total":96636764160,"available":96049545216,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:23.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:23.879 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:23.880 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:23.895 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2024-09-15T21:15:23.620395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690268,"kb_used_data":1308,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797092,"statfs":{"total":386547056640,"available":383792222208,"internally_reserved":0,"allocated":1339392,"data_stored":614236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5636120,"num_objects":-689,"num_object_clones":0,"num_object_copies":-2756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":689,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.410361"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17449,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.340616+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.340616+0000","last_peered":"2024-09-15T21:15:14.340616+0000","last_clean":"2024-09-15T21:15:14.340616+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.340616+0000","last_undegraded":"2024-09-15T21:15:14.340616+0000","last_fullsized":"2024-09-15T21:15:14.340616+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12535,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.350312+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:14.350312+0000","last_peered":"2024-09-15T21:15:14.350312+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:14.350312+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12851,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.204352+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:14.204352+0000","last_peered":"2024-09-15T21:15:14.204352+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:14.204352+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12839,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.365155+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:14.365155+0000","last_peered":"2024-09-15T21:15:14.365155+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:14.365155+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":15452,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:19.166010+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:19.166010+0000","last_peered":"2024-09-15T21:15:19.166010+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:19.166010+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573456,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798384,"statfs":{"total":96636764160,"available":96049545216,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247958,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12352", 'reported_seq': 12409, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.059005+0000', 'last_change': '2024-09-15T21:03:59.666757+0000', 'last_active': '2024-09-15T21:15:14.059005+0000', 'last_peered': '2024-09-15T21:15:14.059005+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:14.059005+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2276, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12352, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -2276}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12523, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.243109+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:14.243109+0000', 'last_peered': '2024-09-15T21:15:14.243109+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:14.243109+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12535, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.350312+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:14.350312+0000', 'last_peered': '2024-09-15T21:15:14.350312+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:14.350312+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:23.896 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12724, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.130081+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:14.130081+0000', 'last_peered': '2024-09-15T21:15:14.130081+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:14.130081+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12432, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.346512+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:14.346512+0000', 'last_peered': '2024-09-15T21:15:14.346512+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:14.346512+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 12851, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.204352+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:14.204352+0000', 'last_peered': '2024-09-15T21:15:14.204352+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:14.204352+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2364}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12839, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.365155+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:14.365155+0000', 'last_peered': '2024-09-15T21:15:14.365155+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:14.365155+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12519, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.144794+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:14.144794+0000', 'last_peered': '2024-09-15T21:15:14.144794+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:14.144794+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12653, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.340319+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:14.340319+0000', 'last_peered': '2024-09-15T21:15:14.340319+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:14.340319+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:23.897 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12477, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.279504+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:14.279504+0000', 'last_peered': '2024-09-15T21:15:14.279504+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:14.279504+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12498, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.202581+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:14.202581+0000', 'last_peered': '2024-09-15T21:15:14.202581+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:14.202581+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12773, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:14.286579+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:14.286579+0000', 'last_peered': '2024-09-15T21:15:14.286579+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:14.286579+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.b is not active+clean 2024-09-15T21:15:23.898 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.b', 'version': "37'12547", 'reported_seq': 15452, 'reported_epoch': 37, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:19.166010+0000', 'last_change': '2024-09-15T21:14:06.196270+0000', 'last_active': '2024-09-15T21:15:19.166010+0000', 'last_peered': '2024-09-15T21:15:19.166010+0000', 'last_clean': '2024-09-15T21:02:49.855171+0000', 'last_became_active': '2024-09-15T21:03:59.654912+0000', 'last_became_peered': '2024-09-15T21:03:59.654912+0000', 'last_unstale': '2024-09-15T21:15:19.166010+0000', 'last_undegraded': '2024-09-15T21:03:59.086850+0000', 'last_fullsized': '2024-09-15T21:03:59.047974+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10047, 'log_dups_size': 0, 'ondisk_log_size': 10047, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 905, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12545, 'num_write_kb': 50188, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1414, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -903}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:23.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:24.361 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:24.361 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 28742/88 objects degraded (32661.364%), 13 pgs degraded, 13 pgs undersized","count":26},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":790,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":12},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22,"data_bytes":149380,"bytes_used":2754834432,"bytes_avail":383792222208,"bytes_total":386547056640,"degraded_objects":28742,"degraded_total":88,"degraded_ratio":326.61363636363637,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":903,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":117},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:24.375 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:24.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:24.776 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:24.777 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:24.789 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-15T21:15:24.621919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690268,"kb_used_data":1308,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797092,"statfs":{"total":386547056640,"available":383792222208,"internally_reserved":0,"allocated":1339392,"data_stored":614236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2793496,"num_objects":-342,"num_object_clones":0,"num_object_copies":-1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-855,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":342,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":855,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.410442"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17449,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.340616+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:14.340616+0000","last_peered":"2024-09-15T21:15:14.340616+0000","last_clean":"2024-09-15T21:15:14.340616+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:14.340616+0000","last_undegraded":"2024-09-15T21:15:14.340616+0000","last_fullsized":"2024-09-15T21:15:14.340616+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.620916+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:23.620916+0000","last_peered":"2024-09-15T21:15:23.620916+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:23.620916+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12853,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621460+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:23.621460+0000","last_peered":"2024-09-15T21:15:23.621460+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:23.621460+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12841,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621499+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:23.621499+0000","last_peered":"2024-09-15T21:15:23.621499+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:23.621499+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17194,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.278481+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:14.278481+0000","last_peered":"2024-09-15T21:15:14.278481+0000","last_clean":"2024-09-15T21:15:14.278481+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:14.278481+0000","last_undegraded":"2024-09-15T21:15:14.278481+0000","last_fullsized":"2024-09-15T21:15:14.278481+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":15452,"reported_epoch":37,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:19.166010+0000","last_change":"2024-09-15T21:14:06.196270+0000","last_active":"2024-09-15T21:15:19.166010+0000","last_peered":"2024-09-15T21:15:19.166010+0000","last_clean":"2024-09-15T21:02:49.855171+0000","last_became_active":"2024-09-15T21:03:59.654912+0000","last_became_peered":"2024-09-15T21:03:59.654912+0000","last_unstale":"2024-09-15T21:15:19.166010+0000","last_undegraded":"2024-09-15T21:03:59.086850+0000","last_fullsized":"2024-09-15T21:03:59.047974+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1414,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-903}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28742,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051720,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573456,"kb_used_data":384,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798384,"statfs":{"total":96636764160,"available":96049545216,"internally_reserved":0,"allocated":393216,"data_stored":211927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:27.791 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:28.114 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:28.114 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:28.128 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-15T21:15:26.213483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698612,"kb_used_data":1324,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788748,"statfs":{"total":386547056640,"available":383783677952,"internally_reserved":0,"allocated":1355776,"data_stored":622359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-131072,"num_objects":-16,"num_object_clones":0,"num_object_copies":-64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1760,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001626"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":12409,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.059005+0000","last_change":"2024-09-15T21:03:59.666757+0000","last_active":"2024-09-15T21:15:14.059005+0000","last_peered":"2024-09-15T21:15:14.059005+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:14.059005+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-2276}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12523,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.243109+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:14.243109+0000","last_peered":"2024-09-15T21:15:14.243109+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:14.243109+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.620916+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:23.620916+0000","last_peered":"2024-09-15T21:15:23.620916+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:23.620916+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12724,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.130081+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:14.130081+0000","last_peered":"2024-09-15T21:15:14.130081+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:14.130081+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12432,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.346512+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:14.346512+0000","last_peered":"2024-09-15T21:15:14.346512+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:14.346512+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12853,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621460+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:23.621460+0000","last_peered":"2024-09-15T21:15:23.621460+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:23.621460+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12841,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621499+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:23.621499+0000","last_peered":"2024-09-15T21:15:23.621499+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:23.621499+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12519,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.144794+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:14.144794+0000","last_peered":"2024-09-15T21:15:14.144794+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:14.144794+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12653,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.340319+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:14.340319+0000","last_peered":"2024-09-15T21:15:14.340319+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:14.340319+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17034,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-15T21:15:14.324734+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:14.324734+0000","last_peered":"2024-09-15T21:15:14.324734+0000","last_clean":"2024-09-15T21:15:14.324734+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:14.324734+0000","last_undegraded":"2024-09-15T21:15:14.324734+0000","last_fullsized":"2024-09-15T21:15:14.324734+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12477,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.279504+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:14.279504+0000","last_peered":"2024-09-15T21:15:14.279504+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:14.279504+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12498,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.202581+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:14.202581+0000","last_peered":"2024-09-15T21:15:14.202581+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:14.202581+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12773,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:14.286579+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:14.286579+0000","last_peered":"2024-09-15T21:15:14.286579+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:14.286579+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13759,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.710784+0000","last_change":"2024-09-15T21:15:24.710135+0000","last_active":"2024-09-15T21:15:24.710784+0000","last_peered":"2024-09-15T21:15:24.710784+0000","last_clean":"2024-09-15T21:15:24.710784+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:24.710784+0000","last_undegraded":"2024-09-15T21:15:24.710784+0000","last_fullsized":"2024-09-15T21:15:24.710784+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27837,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708100,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663740,"statfs":{"total":96636764160,"available":95911669760,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695108,"kb_used_data":308,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676732,"statfs":{"total":96636764160,"available":95924973568,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:28.129 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:28.453 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:28.454 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:28.466 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-15T21:15:28.213881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698660,"kb_used_data":1356,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788700,"statfs":{"total":386547056640,"available":383783628800,"internally_reserved":0,"allocated":1388544,"data_stored":638605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1527,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001642"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":13658,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:26.611916+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:26.611916+0000","last_peered":"2024-09-15T21:15:26.611916+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:26.611916+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":622,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.620916+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:23.620916+0000","last_peered":"2024-09-15T21:15:23.620916+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:23.620916+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12853,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621460+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:23.621460+0000","last_peered":"2024-09-15T21:15:23.621460+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:23.621460+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12841,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621499+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:23.621499+0000","last_peered":"2024-09-15T21:15:23.621499+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:23.621499+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17038,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626786+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626786+0000","last_peered":"2024-09-15T21:15:24.626786+0000","last_clean":"2024-09-15T21:15:24.626786+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626786+0000","last_undegraded":"2024-09-15T21:15:24.626786+0000","last_fullsized":"2024-09-15T21:15:24.626786+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13759,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.710784+0000","last_change":"2024-09-15T21:15:24.710135+0000","last_active":"2024-09-15T21:15:24.710784+0000","last_peered":"2024-09-15T21:15:24.710784+0000","last_clean":"2024-09-15T21:15:24.710784+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:24.710784+0000","last_undegraded":"2024-09-15T21:15:24.710784+0000","last_fullsized":"2024-09-15T21:15:24.710784+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:28.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:28.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12352", 'reported_seq': 13658, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:26.611916+0000', 'last_change': '2024-09-15T21:15:23.051250+0000', 'last_active': '2024-09-15T21:15:26.611916+0000', 'last_peered': '2024-09-15T21:15:26.611916+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:26.611916+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1654, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12352, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 622, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -1654}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.467 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:28.467 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12527, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629926+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:24.629926+0000', 'last_peered': '2024-09-15T21:15:24.629926+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:24.629926+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12537, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:23.620916+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:23.620916+0000', 'last_peered': '2024-09-15T21:15:23.620916+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:23.620916+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12729, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626996+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:24.626996+0000', 'last_peered': '2024-09-15T21:15:24.626996+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:24.626996+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12436, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626251+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:24.626251+0000', 'last_peered': '2024-09-15T21:15:24.626251+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:24.626251+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 12853, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:23.621460+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:23.621460+0000', 'last_peered': '2024-09-15T21:15:23.621460+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:23.621460+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2364}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:28.468 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12841, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:23.621499+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:23.621499+0000', 'last_peered': '2024-09-15T21:15:23.621499+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:23.621499+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.623454+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:24.623454+0000', 'last_peered': '2024-09-15T21:15:24.623454+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:24.623454+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12657, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.630264+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:24.630264+0000', 'last_peered': '2024-09-15T21:15:24.630264+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:24.630264+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12481, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.631449+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:24.631449+0000', 'last_peered': '2024-09-15T21:15:24.631449+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:24.631449+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12502, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.625652+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:24.625652+0000', 'last_peered': '2024-09-15T21:15:24.625652+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:24.625652+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:28.469 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12777, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629732+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:24.629732+0000', 'last_peered': '2024-09-15T21:15:24.629732+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:24.629732+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:28.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:28.850 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:28.850 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27215/88 objects degraded (30926.136%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":794,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":22,"data_bytes":149380,"bytes_used":2763427840,"bytes_avail":383783628800,"bytes_total":386547056640,"degraded_objects":27215,"degraded_total":88,"degraded_ratio":309.26136363636363,"recovering_objects_per_sec":190,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1527,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:28.863 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:28.864 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:29.176 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:29.177 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:29.189 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-15T21:15:28.213881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698660,"kb_used_data":1356,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788700,"statfs":{"total":386547056640,"available":383783628800,"internally_reserved":0,"allocated":1388544,"data_stored":638605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1527,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1527,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001642"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":13658,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:26.611916+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:26.611916+0000","last_peered":"2024-09-15T21:15:26.611916+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:26.611916+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1654,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":622,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-1654}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12537,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.620916+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:23.620916+0000","last_peered":"2024-09-15T21:15:23.620916+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:23.620916+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12853,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621460+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:23.621460+0000","last_peered":"2024-09-15T21:15:23.621460+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:23.621460+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12841,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:23.621499+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:23.621499+0000","last_peered":"2024-09-15T21:15:23.621499+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:23.621499+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17038,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626786+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626786+0000","last_peered":"2024-09-15T21:15:24.626786+0000","last_clean":"2024-09-15T21:15:24.626786+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626786+0000","last_undegraded":"2024-09-15T21:15:24.626786+0000","last_fullsized":"2024-09-15T21:15:24.626786+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13759,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.710784+0000","last_change":"2024-09-15T21:15:24.710135+0000","last_active":"2024-09-15T21:15:24.710784+0000","last_peered":"2024-09-15T21:15:24.710784+0000","last_clean":"2024-09-15T21:15:24.710784+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:24.710784+0000","last_undegraded":"2024-09-15T21:15:24.710784+0000","last_fullsized":"2024-09-15T21:15:24.710784+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12545,"num_write_kb":50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":149380,"num_objects":22,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27215,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173318,"num_write_kb":653908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2960,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247959,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713604,"kb_used_data":308,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658236,"statfs":{"total":96636764160,"available":95906033664,"internally_reserved":0,"allocated":315392,"data_stored":134103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:32.192 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:32.498 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:32.499 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:32.512 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-15T21:15:32.214738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690352,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797008,"statfs":{"total":386547056640,"available":383792136192,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.594343"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:25.660461+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:25.660461+0000","last_peered":"2024-09-15T21:15:25.660461+0000","last_clean":"2024-09-15T21:15:25.660461+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:25.660461+0000","last_undegraded":"2024-09-15T21:15:25.660461+0000","last_fullsized":"2024-09-15T21:15:25.660461+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573472,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798368,"statfs":{"total":96636764160,"available":96049528832,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:32.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:32.836 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:32.837 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:32.849 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-15T21:15:32.214738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690352,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797008,"statfs":{"total":386547056640,"available":383792136192,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.594343"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:25.660461+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:25.660461+0000","last_peered":"2024-09-15T21:15:25.660461+0000","last_clean":"2024-09-15T21:15:25.660461+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:25.660461+0000","last_undegraded":"2024-09-15T21:15:25.660461+0000","last_fullsized":"2024-09-15T21:15:25.660461+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573472,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798368,"statfs":{"total":96636764160,"available":96049528832,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12352", 'reported_seq': 15526, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:31.614206+0000', 'last_change': '2024-09-15T21:15:23.051250+0000', 'last_active': '2024-09-15T21:15:31.614206+0000', 'last_peered': '2024-09-15T21:15:31.614206+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:31.614206+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12352, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1556, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -720}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12527, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629926+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:24.629926+0000', 'last_peered': '2024-09-15T21:15:24.629926+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:24.629926+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12539, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.624257+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:24.624257+0000', 'last_peered': '2024-09-15T21:15:24.624257+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:24.624257+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:32.851 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12729, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626996+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:24.626996+0000', 'last_peered': '2024-09-15T21:15:24.626996+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:24.626996+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12436, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626251+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:24.626251+0000', 'last_peered': '2024-09-15T21:15:24.626251+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:24.626251+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 12855, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.623570+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:24.623570+0000', 'last_peered': '2024-09-15T21:15:24.623570+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:24.623570+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2364}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12843, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.624723+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:24.624723+0000', 'last_peered': '2024-09-15T21:15:24.624723+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:24.624723+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:32.852 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.623454+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:24.623454+0000', 'last_peered': '2024-09-15T21:15:24.623454+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:24.623454+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12657, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.630264+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:24.630264+0000', 'last_peered': '2024-09-15T21:15:24.630264+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:24.630264+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12481, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.631449+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:24.631449+0000', 'last_peered': '2024-09-15T21:15:24.631449+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:24.631449+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:32.853 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12502, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.625652+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:24.625652+0000', 'last_peered': '2024-09-15T21:15:24.625652+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:24.625652+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:32.854 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:32.854 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12777, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629732+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:24.629732+0000', 'last_peered': '2024-09-15T21:15:24.629732+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:24.629732+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:32.854 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:33.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:33.236 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26281/80 objects degraded (32851.250%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":798,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":12},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":135800,"bytes_used":2754920448,"bytes_avail":383792136192,"bytes_total":386547056640,"degraded_objects":26281,"degraded_total":80,"degraded_ratio":328.51249999999999,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:33.249 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:33.249 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:33.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:33.608 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:33.620 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-15T21:15:32.214738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690352,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374797008,"statfs":{"total":386547056640,"available":383792136192,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.594343"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17453,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628412+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:24.628412+0000","last_peered":"2024-09-15T21:15:24.628412+0000","last_clean":"2024-09-15T21:15:24.628412+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:24.628412+0000","last_undegraded":"2024-09-15T21:15:24.628412+0000","last_fullsized":"2024-09-15T21:15:24.628412+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17198,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.628589+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:24.628589+0000","last_peered":"2024-09-15T21:15:24.628589+0000","last_clean":"2024-09-15T21:15:24.628589+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:24.628589+0000","last_undegraded":"2024-09-15T21:15:24.628589+0000","last_fullsized":"2024-09-15T21:15:24.628589+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13767,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:25.660461+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:25.660461+0000","last_peered":"2024-09-15T21:15:25.660461+0000","last_clean":"2024-09-15T21:15:25.660461+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:25.660461+0000","last_undegraded":"2024-09-15T21:15:25.660461+0000","last_fullsized":"2024-09-15T21:15:25.660461+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":94208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051722,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573472,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798368,"statfs":{"total":96636764160,"available":96049528832,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247960,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":81920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:36.623 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:36.947 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:36.948 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:36.960 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-15T21:15:36.288992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698672,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788688,"statfs":{"total":386547056640,"available":383783616512,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075110"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279894+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:35.279894+0000","last_peered":"2024-09-15T21:15:35.279894+0000","last_clean":"2024-09-15T21:15:35.279894+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:35.279894+0000","last_undegraded":"2024-09-15T21:15:35.279894+0000","last_fullsized":"2024-09-15T21:15:35.279894+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17200,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279765+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:35.279765+0000","last_peered":"2024-09-15T21:15:35.279765+0000","last_clean":"2024-09-15T21:15:35.279765+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:35.279765+0000","last_undegraded":"2024-09-15T21:15:35.279765+0000","last_fullsized":"2024-09-15T21:15:35.279765+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13769,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.280015+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:35.280015+0000","last_peered":"2024-09-15T21:15:35.280015+0000","last_clean":"2024-09-15T21:15:35.280015+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:35.280015+0000","last_undegraded":"2024-09-15T21:15:35.280015+0000","last_fullsized":"2024-09-15T21:15:35.280015+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:36.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:37.351 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:37.351 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:37.366 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-15T21:15:36.288992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698672,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788688,"statfs":{"total":386547056640,"available":383783616512,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075110"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279894+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:35.279894+0000","last_peered":"2024-09-15T21:15:35.279894+0000","last_clean":"2024-09-15T21:15:35.279894+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:35.279894+0000","last_undegraded":"2024-09-15T21:15:35.279894+0000","last_fullsized":"2024-09-15T21:15:35.279894+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17200,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279765+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:35.279765+0000","last_peered":"2024-09-15T21:15:35.279765+0000","last_clean":"2024-09-15T21:15:35.279765+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:35.279765+0000","last_undegraded":"2024-09-15T21:15:35.279765+0000","last_fullsized":"2024-09-15T21:15:35.279765+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13769,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.280015+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:35.280015+0000","last_peered":"2024-09-15T21:15:35.280015+0000","last_clean":"2024-09-15T21:15:35.280015+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:35.280015+0000","last_undegraded":"2024-09-15T21:15:35.280015+0000","last_fullsized":"2024-09-15T21:15:35.280015+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.f is not active+clean 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.f', 'version': "37'12352", 'reported_seq': 15526, 'reported_epoch': 39, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:31.614206+0000', 'last_change': '2024-09-15T21:15:23.051250+0000', 'last_active': '2024-09-15T21:15:31.614206+0000', 'last_peered': '2024-09-15T21:15:31.614206+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.657209+0000', 'last_became_peered': '2024-09-15T21:03:59.657209+0000', 'last_unstale': '2024-09-15T21:15:31.614206+0000', 'last_undegraded': '2024-09-15T21:03:59.123275+0000', 'last_fullsized': '2024-09-15T21:03:59.043155+0000', 'mapping_epoch': 30, 'log_start': "31'2300", 'ondisk_log_start': "31'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10052, 'log_dups_size': 0, 'ondisk_log_size': 10052, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 720, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12352, 'num_write_kb': 49408, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1556, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 1, 2], 'acting': [3, 0, 2147483647, 2], 'avail_no_missing': ['3(0)', '0(1)', '2(3)'], 'object_location_counts': [{'shards': '0(1),2(3),3(0)', 'objects': -720}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12527, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629926+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:24.629926+0000', 'last_peered': '2024-09-15T21:15:24.629926+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:24.629926+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12539, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.624257+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:24.624257+0000', 'last_peered': '2024-09-15T21:15:24.624257+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:24.624257+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:37.367 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12729, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626996+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:24.626996+0000', 'last_peered': '2024-09-15T21:15:24.626996+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:24.626996+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12436, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.626251+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:24.626251+0000', 'last_peered': '2024-09-15T21:15:24.626251+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:24.626251+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 12855, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.623570+0000', 'last_change': '2024-09-15T21:03:59.675606+0000', 'last_active': '2024-09-15T21:15:24.623570+0000', 'last_peered': '2024-09-15T21:15:24.623570+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:24.623570+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2368, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -2364}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12843, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.624723+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:24.624723+0000', 'last_peered': '2024-09-15T21:15:24.624723+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:24.624723+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12523, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.623454+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:24.623454+0000', 'last_peered': '2024-09-15T21:15:24.623454+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:24.623454+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:37.368 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12657, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.630264+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:24.630264+0000', 'last_peered': '2024-09-15T21:15:24.630264+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:24.630264+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12481, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.631449+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:24.631449+0000', 'last_peered': '2024-09-15T21:15:24.631449+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:24.631449+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12502, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.625652+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:24.625652+0000', 'last_peered': '2024-09-15T21:15:24.625652+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:24.625652+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:37.369 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12777, 'reported_epoch': 39, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:24.629732+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:24.629732+0000', 'last_peered': '2024-09-15T21:15:24.629732+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:24.629732+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:37.369 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:37.810 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:37.810 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26281/80 objects degraded (32851.250%), 12 pgs degraded, 12 pgs undersized","count":24},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":11},{"state_name":"active+clean","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":135800,"bytes_used":2763440128,"bytes_avail":383783616512,"bytes_total":386547056640,"degraded_objects":26281,"degraded_total":80,"degraded_ratio":328.51249999999999},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:37.823 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:15:37.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:38.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:38.160 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:38.171 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-15T21:15:36.288992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698672,"kb_used_data":1372,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788688,"statfs":{"total":386547056640,"available":383783616512,"internally_reserved":0,"allocated":1404928,"data_stored":646728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12545,"num_write_kb":-50188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1385,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.075110"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":15526,"reported_epoch":39,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:31.614206+0000","last_change":"2024-09-15T21:15:23.051250+0000","last_active":"2024-09-15T21:15:31.614206+0000","last_peered":"2024-09-15T21:15:31.614206+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.657209+0000","last_became_peered":"2024-09-15T21:03:59.657209+0000","last_unstale":"2024-09-15T21:15:31.614206+0000","last_undegraded":"2024-09-15T21:03:59.123275+0000","last_fullsized":"2024-09-15T21:03:59.043155+0000","mapping_epoch":30,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1556,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,2147483647,2],"avail_no_missing":["3(0)","0(1)","2(3)"],"object_location_counts":[{"shards":"0(1),2(3),3(0)","objects":-720}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279894+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:35.279894+0000","last_peered":"2024-09-15T21:15:35.279894+0000","last_clean":"2024-09-15T21:15:35.279894+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:35.279894+0000","last_undegraded":"2024-09-15T21:15:35.279894+0000","last_fullsized":"2024-09-15T21:15:35.279894+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12527,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629926+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:24.629926+0000","last_peered":"2024-09-15T21:15:24.629926+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:24.629926+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624257+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:24.624257+0000","last_peered":"2024-09-15T21:15:24.624257+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:24.624257+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12729,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626996+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:24.626996+0000","last_peered":"2024-09-15T21:15:24.626996+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:24.626996+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12436,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.626251+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:24.626251+0000","last_peered":"2024-09-15T21:15:24.626251+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:24.626251+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":12855,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623570+0000","last_change":"2024-09-15T21:03:59.675606+0000","last_active":"2024-09-15T21:15:24.623570+0000","last_peered":"2024-09-15T21:15:24.623570+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:24.623570+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2368,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-2364}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.624723+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:24.624723+0000","last_peered":"2024-09-15T21:15:24.624723+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:24.624723+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12523,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.623454+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:24.623454+0000","last_peered":"2024-09-15T21:15:24.623454+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:24.623454+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12657,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.630264+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:24.630264+0000","last_peered":"2024-09-15T21:15:24.630264+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:24.630264+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17039,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-15T21:15:24.626851+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:24.626851+0000","last_peered":"2024-09-15T21:15:24.626851+0000","last_clean":"2024-09-15T21:15:24.626851+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:24.626851+0000","last_undegraded":"2024-09-15T21:15:24.626851+0000","last_fullsized":"2024-09-15T21:15:24.626851+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12481,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.631449+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:24.631449+0000","last_peered":"2024-09-15T21:15:24.631449+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:24.631449+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17200,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279765+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:35.279765+0000","last_peered":"2024-09-15T21:15:35.279765+0000","last_clean":"2024-09-15T21:15:35.279765+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:35.279765+0000","last_undegraded":"2024-09-15T21:15:35.279765+0000","last_fullsized":"2024-09-15T21:15:35.279765+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12502,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.625652+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:24.625652+0000","last_peered":"2024-09-15T21:15:24.625652+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:24.625652+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12777,"reported_epoch":39,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:24.629732+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:24.629732+0000","last_peered":"2024-09-15T21:15:24.629732+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:24.629732+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13769,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.280015+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:35.280015+0000","last_peered":"2024-09-15T21:15:35.280015+0000","last_clean":"2024-09-15T21:15:35.280015+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:35.280015+0000","last_undegraded":"2024-09-15T21:15:35.280015+0000","last_fullsized":"2024-09-15T21:15:35.280015+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708124,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663716,"statfs":{"total":96636764160,"available":95911645184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695132,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676708,"statfs":{"total":96636764160,"available":95924948992,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247961,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713624,"kb_used_data":324,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658216,"statfs":{"total":96636764160,"available":95906013184,"internally_reserved":0,"allocated":331776,"data_stored":142226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:41.173 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:41.484 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:41.485 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:41.497 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-15T21:15:40.216311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698712,"kb_used_data":1420,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788648,"statfs":{"total":386547056640,"available":383783575552,"internally_reserved":0,"allocated":1454080,"data_stored":670875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001572"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279894+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:35.279894+0000","last_peered":"2024-09-15T21:15:35.279894+0000","last_clean":"2024-09-15T21:15:35.279894+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:35.279894+0000","last_undegraded":"2024-09-15T21:15:35.279894+0000","last_fullsized":"2024-09-15T21:15:35.279894+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":14563,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:39.169036+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:39.169036+0000","last_peered":"2024-09-15T21:15:39.169036+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:39.169036+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1515}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12847,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296813+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.296813+0000","last_peered":"2024-09-15T21:15:36.296813+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.296813+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17200,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279765+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:35.279765+0000","last_peered":"2024-09-15T21:15:35.279765+0000","last_clean":"2024-09-15T21:15:35.279765+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:35.279765+0000","last_undegraded":"2024-09-15T21:15:35.279765+0000","last_fullsized":"2024-09-15T21:15:35.279765+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13769,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.280015+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:35.280015+0000","last_peered":"2024-09-15T21:15:35.280015+0000","last_clean":"2024-09-15T21:15:35.280015+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:35.280015+0000","last_undegraded":"2024-09-15T21:15:35.280015+0000","last_fullsized":"2024-09-15T21:15:35.280015+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:41.498 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:41.814 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:41.814 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:41.827 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-15T21:15:40.216311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698712,"kb_used_data":1420,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788648,"statfs":{"total":386547056640,"available":383783575552,"internally_reserved":0,"allocated":1454080,"data_stored":670875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001572"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17455,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279894+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:35.279894+0000","last_peered":"2024-09-15T21:15:35.279894+0000","last_clean":"2024-09-15T21:15:35.279894+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:35.279894+0000","last_undegraded":"2024-09-15T21:15:35.279894+0000","last_fullsized":"2024-09-15T21:15:35.279894+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":14563,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:39.169036+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:39.169036+0000","last_peered":"2024-09-15T21:15:39.169036+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:39.169036+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1515}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12847,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296813+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.296813+0000","last_peered":"2024-09-15T21:15:36.296813+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.296813+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17200,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.279765+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:35.279765+0000","last_peered":"2024-09-15T21:15:35.279765+0000","last_clean":"2024-09-15T21:15:35.279765+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:35.279765+0000","last_undegraded":"2024-09-15T21:15:35.279765+0000","last_fullsized":"2024-09-15T21:15:35.279765+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13769,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-15T21:15:35.280015+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:35.280015+0000","last_peered":"2024-09-15T21:15:35.280015+0000","last_clean":"2024-09-15T21:15:35.280015+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:35.280015+0000","last_undegraded":"2024-09-15T21:15:35.280015+0000","last_fullsized":"2024-09-15T21:15:35.280015+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051723,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581792,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790048,"statfs":{"total":96636764160,"available":96041009152,"internally_reserved":0,"allocated":409600,"data_stored":220050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12531, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290248+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:36.290248+0000', 'last_peered': '2024-09-15T21:15:36.290248+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:36.290248+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12543, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.296477+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:36.296477+0000', 'last_peered': '2024-09-15T21:15:36.296477+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:36.296477+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12733, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293223+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:36.293223+0000', 'last_peered': '2024-09-15T21:15:36.293223+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:36.293223+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12440, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293901+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:36.293901+0000', 'last_peered': '2024-09-15T21:15:36.293901+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:36.293901+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:41.828 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 14563, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:39.169036+0000', 'last_change': '2024-09-15T21:15:34.795999+0000', 'last_active': '2024-09-15T21:15:39.169036+0000', 'last_peered': '2024-09-15T21:15:39.169036+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:39.169036+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1517, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 853, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -1515}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12847, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.296813+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:36.296813+0000', 'last_peered': '2024-09-15T21:15:36.296813+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:36.296813+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12527, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293719+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:36.293719+0000', 'last_peered': '2024-09-15T21:15:36.293719+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:36.293719+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12661, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290470+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:36.290470+0000', 'last_peered': '2024-09-15T21:15:36.290470+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:36.290470+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:41.829 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12485, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293693+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:36.293693+0000', 'last_peered': '2024-09-15T21:15:36.293693+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:36.293693+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:41.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:41.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12506, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293475+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:36.293475+0000', 'last_peered': '2024-09-15T21:15:36.293475+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:36.293475+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:41.830 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:41.830 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290159+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:36.290159+0000', 'last_peered': '2024-09-15T21:15:36.290159+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:36.290159+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:41.830 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:42.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:42.201 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 24710/80 objects degraded (30887.500%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":807,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":135800,"bytes_used":2763481088,"bytes_avail":383783575552,"bytes_total":386547056640,"degraded_objects":24710,"degraded_total":80,"degraded_ratio":308.875,"recovering_objects_per_sec":196,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:42.217 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:42.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:42.587 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:42.587 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:42.600 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":436,"stamp":"2024-09-15T21:15:42.216656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690400,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374796960,"statfs":{"total":386547056640,"available":383792087040,"internally_reserved":0,"allocated":1462272,"data_stored":670732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1571,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1571,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001514"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":14563,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:39.169036+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:39.169036+0000","last_peered":"2024-09-15T21:15:39.169036+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:39.169036+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1517,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":853,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-1515}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12847,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296813+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.296813+0000","last_peered":"2024-09-15T21:15:36.296813+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.296813+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24710,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3146,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573480,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798360,"statfs":{"total":96636764160,"available":96049520640,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247962,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:45.602 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:45.919 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:45.920 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:45.932 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-15T21:15:44.217091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690400,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374796960,"statfs":{"total":386547056640,"available":383792087040,"internally_reserved":0,"allocated":1462272,"data_stored":670732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.939708"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":16451,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:44.171161+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:44.171161+0000","last_peered":"2024-09-15T21:15:44.171161+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:44.171161+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1797,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-571}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12848,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.298375+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.298375+0000","last_peered":"2024-09-15T21:15:36.298375+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.298375+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051724,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573480,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798360,"statfs":{"total":96636764160,"available":96049520640,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:45.934 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:46.267 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:46.267 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:46.280 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-15T21:15:46.217463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698720,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788640,"statfs":{"total":386547056640,"available":383783567360,"internally_reserved":0,"allocated":1462272,"data_stored":670732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001969"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":16451,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:44.171161+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:44.171161+0000","last_peered":"2024-09-15T21:15:44.171161+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:44.171161+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1797,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-571}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12848,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.298375+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.298375+0000","last_peered":"2024-09-15T21:15:36.298375+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.298375+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:46.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:46.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12531, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290248+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:36.290248+0000', 'last_peered': '2024-09-15T21:15:36.290248+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:36.290248+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:46.281 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:46.281 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12543, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.296477+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:36.296477+0000', 'last_peered': '2024-09-15T21:15:36.296477+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:36.296477+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12733, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293223+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:36.293223+0000', 'last_peered': '2024-09-15T21:15:36.293223+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:36.293223+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12440, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293901+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:36.293901+0000', 'last_peered': '2024-09-15T21:15:36.293901+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:36.293901+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.2 is not active+clean 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.2', 'version': "37'12770", 'reported_seq': 16451, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:44.171161+0000', 'last_change': '2024-09-15T21:15:34.795999+0000', 'last_active': '2024-09-15T21:15:44.171161+0000', 'last_peered': '2024-09-15T21:15:44.171161+0000', 'last_clean': '2024-09-15T21:02:50.776500+0000', 'last_became_active': '2024-09-15T21:03:59.663436+0000', 'last_became_peered': '2024-09-15T21:03:59.663436+0000', 'last_unstale': '2024-09-15T21:15:44.171161+0000', 'last_undegraded': '2024-09-15T21:03:59.092087+0000', 'last_fullsized': '2024-09-15T21:03:59.044281+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10070, 'log_dups_size': 0, 'ondisk_log_size': 10070, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 573, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12768, 'num_write_kb': 51076, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1797, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3, 1, 2], 'acting': [0, 3, 2147483647, 2], 'avail_no_missing': ['0(0)', '2(3)', '3(1)'], 'object_location_counts': [{'shards': '0(0),2(3),3(1)', 'objects': -571}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12848, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.298375+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:36.298375+0000', 'last_peered': '2024-09-15T21:15:36.298375+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:36.298375+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:46.282 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12527, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293719+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:36.293719+0000', 'last_peered': '2024-09-15T21:15:36.293719+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:36.293719+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12661, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290470+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:36.290470+0000', 'last_peered': '2024-09-15T21:15:36.290470+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:36.290470+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12485, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293693+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:36.293693+0000', 'last_peered': '2024-09-15T21:15:36.293693+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:36.293693+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12506, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293475+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:36.293475+0000', 'last_peered': '2024-09-15T21:15:36.293475+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:36.293475+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:46.283 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12781, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290159+0000', 'last_change': '2024-09-15T21:03:59.711923+0000', 'last_active': '2024-09-15T21:15:36.290159+0000', 'last_peered': '2024-09-15T21:15:36.290159+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:36.290159+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2283, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2279}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:46.283 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:46.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:46.702 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23766/80 objects degraded (29707.500%), 11 pgs degraded, 11 pgs undersized","count":22},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":812,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":11},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":10},{"state_name":"active+clean","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":20,"data_bytes":135800,"bytes_used":2763489280,"bytes_avail":383783567360,"bytes_total":386547056640,"degraded_objects":23766,"degraded_total":80,"degraded_ratio":297.07499999999999,"recovering_objects_per_sec":251,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:46.716 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:15:46.716 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:47.016 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:47.017 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:47.030 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":438,"stamp":"2024-09-15T21:15:46.217463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698720,"kb_used_data":1428,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788640,"statfs":{"total":386547056640,"available":383783567360,"internally_reserved":0,"allocated":1462272,"data_stored":670732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2515,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001969"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12543,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.296477+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:36.296477+0000","last_peered":"2024-09-15T21:15:36.296477+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:36.296477+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":16451,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:44.171161+0000","last_change":"2024-09-15T21:15:34.795999+0000","last_active":"2024-09-15T21:15:44.171161+0000","last_peered":"2024-09-15T21:15:44.171161+0000","last_clean":"2024-09-15T21:02:50.776500+0000","last_became_active":"2024-09-15T21:03:59.663436+0000","last_became_peered":"2024-09-15T21:03:59.663436+0000","last_unstale":"2024-09-15T21:15:44.171161+0000","last_undegraded":"2024-09-15T21:03:59.092087+0000","last_fullsized":"2024-09-15T21:03:59.044281+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12768,"num_write_kb":51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1797,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,2147483647,2],"avail_no_missing":["0(0)","2(3)","3(1)"],"object_location_counts":[{"shards":"0(0),2(3),3(1)","objects":-571}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12848,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.298375+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:36.298375+0000","last_peered":"2024-09-15T21:15:36.298375+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:36.298375+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12781,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290159+0000","last_change":"2024-09-15T21:03:59.711923+0000","last_active":"2024-09-15T21:15:36.290159+0000","last_peered":"2024-09-15T21:15:36.290159+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:36.290159+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2279}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":135800,"num_objects":20,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23766,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":160773,"num_write_kb":603720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4090,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247963,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713640,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658200,"statfs":{"total":96636764160,"available":95905996800,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:50.032 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:50.356 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:50.357 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:50.370 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-15T21:15:50.218217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698736,"kb_used_data":1444,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788624,"statfs":{"total":386547056640,"available":383783550976,"internally_reserved":0,"allocated":1478656,"data_stored":678707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-770,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001561"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12948,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:47.102289+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:47.102289+0000","last_peered":"2024-09-15T21:15:47.102289+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:47.102289+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:50.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:50.702 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:50.702 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:50.715 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-15T21:15:50.218217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698736,"kb_used_data":1444,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788624,"statfs":{"total":386547056640,"available":383783550976,"internally_reserved":0,"allocated":1478656,"data_stored":678707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-770,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001561"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12948,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:47.102289+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:47.102289+0000","last_peered":"2024-09-15T21:15:47.102289+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:47.102289+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12531, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290248+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:36.290248+0000', 'last_peered': '2024-09-15T21:15:36.290248+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:36.290248+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12547, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329308+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:48.329308+0000', 'last_peered': '2024-09-15T21:15:48.329308+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:48.329308+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:50.716 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12733, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293223+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:36.293223+0000', 'last_peered': '2024-09-15T21:15:36.293223+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:36.293223+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12440, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293901+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:36.293901+0000', 'last_peered': '2024-09-15T21:15:36.293901+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:36.293901+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12852, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329959+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:48.329959+0000', 'last_peered': '2024-09-15T21:15:48.329959+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:48.329959+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12527, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293719+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:36.293719+0000', 'last_peered': '2024-09-15T21:15:36.293719+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:36.293719+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12661, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.290470+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:36.290470+0000', 'last_peered': '2024-09-15T21:15:36.290470+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:36.290470+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:50.717 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:50.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12485, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293693+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:36.293693+0000', 'last_peered': '2024-09-15T21:15:36.293693+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:36.293693+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:50.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:50.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12506, 'reported_epoch': 41, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:36.293475+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:36.293475+0000', 'last_peered': '2024-09-15T21:15:36.293475+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:36.293475+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:50.718 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:50.718 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 12948, 'reported_epoch': 41, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:47.102289+0000', 'last_change': '2024-09-15T21:15:46.463423+0000', 'last_active': '2024-09-15T21:15:47.102289+0000', 'last_peered': '2024-09-15T21:15:47.102289+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:47.102289+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 85, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -2198}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:50.718 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:51.096 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:51.096 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 23110/72 objects degraded (32097.222%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":816,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":122220,"bytes_used":2763505664,"bytes_avail":383783550976,"bytes_total":386547056640,"degraded_objects":23110,"degraded_total":72,"degraded_ratio":320.97222222222223},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:51.112 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:15:51.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:51.419 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:51.419 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:51.432 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-15T21:15:50.218217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698736,"kb_used_data":1444,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788624,"statfs":{"total":386547056640,"available":383783550976,"internally_reserved":0,"allocated":1478656,"data_stored":678707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1600,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-770,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001561"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16979,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.377056+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:36.377056+0000","last_peered":"2024-09-15T21:15:36.377056+0000","last_clean":"2024-09-15T21:15:36.377056+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:36.377056+0000","last_undegraded":"2024-09-15T21:15:36.377056+0000","last_fullsized":"2024-09-15T21:15:36.377056+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17457,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298693+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:36.298693+0000","last_peered":"2024-09-15T21:15:36.298693+0000","last_clean":"2024-09-15T21:15:36.298693+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:36.298693+0000","last_undegraded":"2024-09-15T21:15:36.298693+0000","last_fullsized":"2024-09-15T21:15:36.298693+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12531,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290248+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:36.290248+0000","last_peered":"2024-09-15T21:15:36.290248+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:36.290248+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12733,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293223+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:36.293223+0000","last_peered":"2024-09-15T21:15:36.293223+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:36.293223+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12440,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293901+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:36.293901+0000","last_peered":"2024-09-15T21:15:36.293901+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:36.293901+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12527,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293719+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:36.293719+0000","last_peered":"2024-09-15T21:15:36.293719+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:36.293719+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12661,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.290470+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:36.290470+0000","last_peered":"2024-09-15T21:15:36.290470+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:36.290470+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17043,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.294652+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:36.294652+0000","last_peered":"2024-09-15T21:15:36.294652+0000","last_clean":"2024-09-15T21:15:36.294652+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:36.294652+0000","last_undegraded":"2024-09-15T21:15:36.294652+0000","last_fullsized":"2024-09-15T21:15:36.294652+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12485,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293693+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:36.293693+0000","last_peered":"2024-09-15T21:15:36.293693+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:36.293693+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17202,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298144+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:36.298144+0000","last_peered":"2024-09-15T21:15:36.298144+0000","last_clean":"2024-09-15T21:15:36.298144+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:36.298144+0000","last_undegraded":"2024-09-15T21:15:36.298144+0000","last_fullsized":"2024-09-15T21:15:36.298144+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12506,"reported_epoch":41,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:36.293475+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:36.293475+0000","last_peered":"2024-09-15T21:15:36.293475+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:36.293475+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12948,"reported_epoch":41,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:47.102289+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:47.102289+0000","last_peered":"2024-09-15T21:15:47.102289+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:47.102289+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-2198}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13771,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-15T21:15:36.298471+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:36.298471+0000","last_peered":"2024-09-15T21:15:36.298471+0000","last_clean":"2024-09-15T21:15:36.298471+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:36.298471+0000","last_undegraded":"2024-09-15T21:15:36.298471+0000","last_fullsized":"2024-09-15T21:15:36.298471+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2376,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708136,"kb_used_data":340,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663704,"statfs":{"total":96636764160,"available":95911632896,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695144,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676696,"statfs":{"total":96636764160,"available":95924936704,"internally_reserved":0,"allocated":348160,"data_stored":150275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051725,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581800,"kb_used_data":408,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790040,"statfs":{"total":96636764160,"available":96041000960,"internally_reserved":0,"allocated":417792,"data_stored":219907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247964,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":73728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:54.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:54.758 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:54.758 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:54.770 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-15T21:15:54.218941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694624,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2693041,"kb_avail":374792736,"statfs":{"total":386547056640,"available":383787761664,"internally_reserved":0,"allocated":1519616,"data_stored":694440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2757674546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":14710,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:52.102073+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:52.102073+0000","last_peered":"2024-09-15T21:15:52.102073+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:52.102073+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577652,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794188,"statfs":{"total":96636764160,"available":96045248512,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:54.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:55.080 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:55.080 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:55.094 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-15T21:15:54.218941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694624,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2693041,"kb_avail":374792736,"statfs":{"total":386547056640,"available":383787761664,"internally_reserved":0,"allocated":1519616,"data_stored":694440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2757674546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":14710,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:52.102073+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:52.102073+0000","last_peered":"2024-09-15T21:15:52.102073+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:52.102073+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577652,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794188,"statfs":{"total":96636764160,"available":96045248512,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12535, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.333060+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:48.333060+0000', 'last_peered': '2024-09-15T21:15:48.333060+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:48.333060+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12547, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329308+0000', 'last_change': '2024-09-15T21:03:59.712197+0000', 'last_active': '2024-09-15T21:15:48.329308+0000', 'last_peered': '2024-09-15T21:15:48.329308+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:48.329308+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2336, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2332}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12737, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.326849+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:48.326849+0000', 'last_peered': '2024-09-15T21:15:48.326849+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:48.326849+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:55.095 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12444, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329249+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:48.329249+0000', 'last_peered': '2024-09-15T21:15:48.329249+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:48.329249+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12852, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329959+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:48.329959+0000', 'last_peered': '2024-09-15T21:15:48.329959+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:48.329959+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12532, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.328543+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:48.328543+0000', 'last_peered': '2024-09-15T21:15:48.328543+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:48.328543+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12665, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.333161+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:48.333161+0000', 'last_peered': '2024-09-15T21:15:48.333161+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:48.333161+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12489, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.326605+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:48.326605+0000', 'last_peered': '2024-09-15T21:15:48.326605+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:48.326605+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12510, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.328747+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:48.328747+0000', 'last_peered': '2024-09-15T21:15:48.328747+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:48.328747+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:55.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:55.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 14710, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:52.102073+0000', 'last_change': '2024-09-15T21:15:46.463423+0000', 'last_active': '2024-09-15T21:15:52.102073+0000', 'last_peered': '2024-09-15T21:15:52.102073+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:52.102073+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1321, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 964, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -1319}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:55.097 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:15:55.486 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:55.486 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22231/72 objects degraded (30876.389%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":821,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":10},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":122220,"bytes_used":2759294976,"bytes_avail":383787761664,"bytes_total":386547056640,"degraded_objects":22231,"degraded_total":72,"degraded_ratio":308.76388888888891},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:15:55.500 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:15:55.500 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:55.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:55.832 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:55.845 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-15T21:15:54.218941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2694624,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2693041,"kb_avail":374792736,"statfs":{"total":386547056640,"available":383787761664,"internally_reserved":0,"allocated":1519616,"data_stored":694440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2757674546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1535,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-835,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001478"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":14710,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:52.102073+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:52.102073+0000","last_peered":"2024-09-15T21:15:52.102073+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:52.102073+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":964,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-1319}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051726,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":577652,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":577211,"kb_avail":93794188,"statfs":{"total":96636764160,"available":96045248512,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":591064783},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:58.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:59.196 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:59.196 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:59.211 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-15T21:15:58.344807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4335,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698780,"kb_used_data":1484,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788580,"statfs":{"total":386547056640,"available":383783505920,"internally_reserved":0,"allocated":1519616,"data_stored":694440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12768,"num_write_kb":-51076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.020021"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12547,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329308+0000","last_change":"2024-09-15T21:03:59.712197+0000","last_active":"2024-09-15T21:15:48.329308+0000","last_peered":"2024-09-15T21:15:48.329308+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:48.329308+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2336,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2332}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17618,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:49.080697+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:49.080697+0000","last_peered":"2024-09-15T21:15:49.080697+0000","last_clean":"2024-09-15T21:15:49.080697+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:49.080697+0000","last_undegraded":"2024-09-15T21:15:49.080697+0000","last_fullsized":"2024-09-15T21:15:49.080697+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12852,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329959+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:48.329959+0000","last_peered":"2024-09-15T21:15:48.329959+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:48.329959+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":16870,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:57.102897+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:57.102897+0000","last_peered":"2024-09-15T21:15:57.102897+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:57.102897+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4335,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581808,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790032,"statfs":{"total":96636764160,"available":96040992768,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247965,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713656,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658184,"statfs":{"total":96636764160,"available":95905980416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:59.212 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:15:59.573 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:15:59.574 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:15:59.586 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-15T21:15:59.348099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698788,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788572,"statfs":{"total":386547056640,"available":383783497728,"internally_reserved":0,"allocated":1527808,"data_stored":698401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.129882"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":13056,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.175115+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:15:59.175115+0000","last_peered":"2024-09-15T21:15:59.175115+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:59.175115+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T21:15:58.349808+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:58.349808+0000","last_peered":"2024-09-15T21:15:58.349808+0000","last_clean":"2024-09-15T21:15:58.349808+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:58.349808+0000","last_undegraded":"2024-09-15T21:15:58.349808+0000","last_fullsized":"2024-09-15T21:15:58.349808+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12854,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:58.348086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:58.348086+0000","last_peered":"2024-09-15T21:15:58.348086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:58.348086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":16870,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:57.102897+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:57.102897+0000","last_peered":"2024-09-15T21:15:57.102897+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:57.102897+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581808,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790032,"statfs":{"total":96636764160,"available":96040992768,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":372736,"data_stored":162211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:15:59.587 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:15:59.587 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12535, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.333060+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:48.333060+0000', 'last_peered': '2024-09-15T21:15:48.333060+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:48.333060+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 13056, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.175115+0000', 'last_change': '2024-09-15T21:15:57.824583+0000', 'last_active': '2024-09-15T21:15:59.175115+0000', 'last_peered': '2024-09-15T21:15:59.175115+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:59.175115+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 255, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2081}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12737, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.326849+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:48.326849+0000', 'last_peered': '2024-09-15T21:15:48.326849+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:48.326849+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12444, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.329249+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:48.329249+0000', 'last_peered': '2024-09-15T21:15:48.329249+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:48.329249+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12854, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:58.348086+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:58.348086+0000', 'last_peered': '2024-09-15T21:15:58.348086+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:58.348086+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:15:59.588 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12532, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.328543+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:48.328543+0000', 'last_peered': '2024-09-15T21:15:48.328543+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:48.328543+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12665, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.333161+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:48.333161+0000', 'last_peered': '2024-09-15T21:15:48.333161+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:48.333161+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12489, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.326605+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:48.326605+0000', 'last_peered': '2024-09-15T21:15:48.326605+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:48.326605+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12510, 'reported_epoch': 43, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:48.328747+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:48.328747+0000', 'last_peered': '2024-09-15T21:15:48.328747+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:48.328747+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.a is not active+clean 2024-09-15T21:15:59.589 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.a', 'version': "37'12679", 'reported_seq': 16870, 'reported_epoch': 43, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:57.102897+0000', 'last_change': '2024-09-15T21:15:46.463423+0000', 'last_active': '2024-09-15T21:15:57.102897+0000', 'last_peered': '2024-09-15T21:15:57.102897+0000', 'last_clean': '2024-09-15T21:02:50.671526+0000', 'last_became_active': '2024-09-15T21:03:59.704037+0000', 'last_became_peered': '2024-09-15T21:03:59.704037+0000', 'last_unstale': '2024-09-15T21:15:57.102897+0000', 'last_undegraded': '2024-09-15T21:03:59.084099+0000', 'last_fullsized': '2024-09-15T21:03:59.040899+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10079, 'log_dups_size': 0, 'ondisk_log_size': 10079, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 241, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12677, 'num_write_kb': 50717, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2044, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2, 3, 0], 'acting': [2147483647, 2, 3, 0], 'avail_no_missing': ['2(1)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(1),3(2)', 'objects': -239}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:15:59.589 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:00.014 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:00.014 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 21151/72 objects degraded (29376.389%), 10 pgs degraded, 10 pgs undersized","count":20},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":825,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":9},{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":18,"data_bytes":122220,"bytes_used":2763550720,"bytes_avail":383783505920,"bytes_total":386547056640,"degraded_objects":21151,"degraded_total":72,"degraded_ratio":293.76388888888891,"recovering_objects_per_sec":16,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":162,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:00.027 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:16:00.028 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:00.330 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:00.330 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:00.344 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":449,"stamp":"2024-09-15T21:16:00.220000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2698788,"kb_used_data":1492,"kb_used_omap":14,"kb_used_meta":2697201,"kb_avail":374788572,"statfs":{"total":386547056640,"available":383783497728,"internally_reserved":0,"allocated":1527808,"data_stored":698401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2761934386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001451"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16983,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326493+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:48.326493+0000","last_peered":"2024-09-15T21:15:48.326493+0000","last_clean":"2024-09-15T21:15:48.326493+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:48.326493+0000","last_undegraded":"2024-09-15T21:15:48.326493+0000","last_fullsized":"2024-09-15T21:15:48.326493+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17461,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326982+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:48.326982+0000","last_peered":"2024-09-15T21:15:48.326982+0000","last_clean":"2024-09-15T21:15:48.326982+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:48.326982+0000","last_undegraded":"2024-09-15T21:15:48.326982+0000","last_fullsized":"2024-09-15T21:15:48.326982+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12535,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333060+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:48.333060+0000","last_peered":"2024-09-15T21:15:48.333060+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:48.333060+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":13056,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.175115+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:15:59.175115+0000","last_peered":"2024-09-15T21:15:59.175115+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:59.175115+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12737,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326849+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:48.326849+0000","last_peered":"2024-09-15T21:15:48.326849+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:48.326849+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12444,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.329249+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:48.329249+0000","last_peered":"2024-09-15T21:15:48.329249+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:48.329249+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T21:15:58.349808+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:58.349808+0000","last_peered":"2024-09-15T21:15:58.349808+0000","last_clean":"2024-09-15T21:15:58.349808+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:58.349808+0000","last_undegraded":"2024-09-15T21:15:58.349808+0000","last_fullsized":"2024-09-15T21:15:58.349808+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12854,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:58.348086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:58.348086+0000","last_peered":"2024-09-15T21:15:58.348086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:58.348086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12532,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328543+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:48.328543+0000","last_peered":"2024-09-15T21:15:48.328543+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:48.328543+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12665,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.333161+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:48.333161+0000","last_peered":"2024-09-15T21:15:48.333161+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:48.333161+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17047,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326632+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:48.326632+0000","last_peered":"2024-09-15T21:15:48.326632+0000","last_clean":"2024-09-15T21:15:48.326632+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:48.326632+0000","last_undegraded":"2024-09-15T21:15:48.326632+0000","last_fullsized":"2024-09-15T21:15:48.326632+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12489,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.326605+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:48.326605+0000","last_peered":"2024-09-15T21:15:48.326605+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:48.326605+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17206,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326124+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:48.326124+0000","last_peered":"2024-09-15T21:15:48.326124+0000","last_clean":"2024-09-15T21:15:48.326124+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:48.326124+0000","last_undegraded":"2024-09-15T21:15:48.326124+0000","last_fullsized":"2024-09-15T21:15:48.326124+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12510,"reported_epoch":43,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:48.328747+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:48.328747+0000","last_peered":"2024-09-15T21:15:48.328747+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:48.328747+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":16870,"reported_epoch":43,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:57.102897+0000","last_change":"2024-09-15T21:15:46.463423+0000","last_active":"2024-09-15T21:15:57.102897+0000","last_peered":"2024-09-15T21:15:57.102897+0000","last_clean":"2024-09-15T21:02:50.671526+0000","last_became_active":"2024-09-15T21:03:59.704037+0000","last_became_peered":"2024-09-15T21:03:59.704037+0000","last_unstale":"2024-09-15T21:15:57.102897+0000","last_undegraded":"2024-09-15T21:03:59.084099+0000","last_fullsized":"2024-09-15T21:03:59.040899+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[2147483647,2,3,0],"avail_no_missing":["2(1)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(1),3(2)","objects":-239}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13775,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-15T21:15:48.326695+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:48.326695+0000","last_peered":"2024-09-15T21:15:48.326695+0000","last_clean":"2024-09-15T21:15:48.326695+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:48.326695+0000","last_undegraded":"2024-09-15T21:15:48.326695+0000","last_fullsized":"2024-09-15T21:15:48.326695+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":122220,"num_objects":18,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148005,"num_write_kb":552644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":77824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708156,"kb_used_data":356,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663684,"statfs":{"total":96636764160,"available":95911612416,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695160,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676680,"statfs":{"total":96636764160,"available":95924920320,"internally_reserved":0,"allocated":364544,"data_stored":158250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051727,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581808,"kb_used_data":416,"kb_used_omap":4,"kb_used_meta":581371,"kb_avail":93790032,"statfs":{"total":96636764160,"available":96040992768,"internally_reserved":0,"allocated":425984,"data_stored":219690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":595324623},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":372736,"data_stored":162211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":65536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:03.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:03.645 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:03.645 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:03.657 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-15T21:16:02.220368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690508,"kb_used_data":1532,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374796852,"statfs":{"total":386547056640,"available":383791976448,"internally_reserved":0,"allocated":1568768,"data_stored":713912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-711,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001427"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":13056,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.175115+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:15:59.175115+0000","last_peered":"2024-09-15T21:15:59.175115+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:59.175115+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T21:15:58.349808+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:58.349808+0000","last_peered":"2024-09-15T21:15:58.349808+0000","last_clean":"2024-09-15T21:15:58.349808+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:58.349808+0000","last_undegraded":"2024-09-15T21:15:58.349808+0000","last_fullsized":"2024-09-15T21:15:58.349808+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12854,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:58.348086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:58.348086+0000","last_peered":"2024-09-15T21:15:58.348086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:58.348086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573496,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798344,"statfs":{"total":96636764160,"available":96049504256,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":372736,"data_stored":162211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:03.658 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:04.013 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:04.014 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:04.027 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-15T21:16:02.220368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690508,"kb_used_data":1532,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374796852,"statfs":{"total":386547056640,"available":383791976448,"internally_reserved":0,"allocated":1568768,"data_stored":713912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1574,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-711,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001427"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":13056,"reported_epoch":44,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.175115+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:15:59.175115+0000","last_peered":"2024-09-15T21:15:59.175115+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:15:59.175115+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2083,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2081}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17620,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-15T21:15:58.349808+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:58.349808+0000","last_peered":"2024-09-15T21:15:58.349808+0000","last_clean":"2024-09-15T21:15:58.349808+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:58.349808+0000","last_undegraded":"2024-09-15T21:15:58.349808+0000","last_fullsized":"2024-09-15T21:15:58.349808+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12854,"reported_epoch":44,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:58.348086+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:58.348086+0000","last_peered":"2024-09-15T21:15:58.348086+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:58.348086+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573496,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798344,"statfs":{"total":96636764160,"available":96049504256,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247966,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713664,"kb_used_data":364,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658176,"statfs":{"total":96636764160,"available":95905972224,"internally_reserved":0,"allocated":372736,"data_stored":162211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:04.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:04.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12539, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.353044+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:59.353044+0000', 'last_peered': '2024-09-15T21:15:59.353044+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:59.353044+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:04.028 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:16:04.028 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 13056, 'reported_epoch': 44, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.175115+0000', 'last_change': '2024-09-15T21:15:57.824583+0000', 'last_active': '2024-09-15T21:15:59.175115+0000', 'last_peered': '2024-09-15T21:15:59.175115+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:15:59.175115+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2083, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 255, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2081}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12741, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.348561+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:59.348561+0000', 'last_peered': '2024-09-15T21:15:59.348561+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:59.348561+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12448, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.350830+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:59.350830+0000', 'last_peered': '2024-09-15T21:15:59.350830+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:59.350830+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12854, 'reported_epoch': 44, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:58.348086+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:58.348086+0000', 'last_peered': '2024-09-15T21:15:58.348086+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:58.348086+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12536, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.350421+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:59.350421+0000', 'last_peered': '2024-09-15T21:15:59.350421+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:59.350421+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12669, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.354834+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:59.354834+0000', 'last_peered': '2024-09-15T21:15:59.354834+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:59.354834+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:04.029 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:04.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12493, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.351047+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:59.351047+0000', 'last_peered': '2024-09-15T21:15:59.351047+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:59.351047+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:04.030 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:04.030 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12514, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.348646+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:59.348646+0000', 'last_peered': '2024-09-15T21:15:59.348646+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:59.348646+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:04.030 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:04.449 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:04.449 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19849/64 objects degraded (31014.062%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":108640,"bytes_used":2755092480,"bytes_avail":383791964160,"bytes_total":386547056640,"degraded_objects":19849,"degraded_total":64,"degraded_ratio":310.140625,"recovering_objects_per_sec":12,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:04.463 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:16:04.463 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:04.777 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:04.778 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:04.790 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":451,"stamp":"2024-09-15T21:16:04.220720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2690520,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2688881,"kb_avail":374796840,"statfs":{"total":386547056640,"available":383791964160,"internally_reserved":0,"allocated":1576960,"data_stored":717852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2753414706},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2382,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001420"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":14674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:04.173844+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:16:04.173844+0000","last_peered":"2024-09-15T21:16:04.173844+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:04.173844+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051728,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":573496,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":573051,"kb_avail":93798344,"statfs":{"total":96636764160,"available":96049504256,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":586804943},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:07.792 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:08.123 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:08.135 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-15T21:16:06.221058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716888,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770472,"statfs":{"total":386547056640,"available":383764963328,"internally_reserved":0,"allocated":1576960,"data_stored":717852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-983,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001426"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":14674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:04.173844+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:16:04.173844+0000","last_peered":"2024-09-15T21:16:04.173844+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:04.173844+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:08.136 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:08.437 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:08.438 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:08.450 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-15T21:16:08.221389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716888,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770472,"statfs":{"total":386547056640,"available":383764963328,"internally_reserved":0,"allocated":1576960,"data_stored":717852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-983,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.876582"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":14674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:04.173844+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:16:04.173844+0000","last_peered":"2024-09-15T21:16:04.173844+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:04.173844+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12539, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.353044+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:15:59.353044+0000', 'last_peered': '2024-09-15T21:15:59.353044+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:15:59.353044+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 14674, 'reported_epoch': 45, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:04.173844+0000', 'last_change': '2024-09-15T21:15:57.824583+0000', 'last_active': '2024-09-15T21:16:04.173844+0000', 'last_peered': '2024-09-15T21:16:04.173844+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:16:04.173844+0000', 'last_undegraded': '2024-09-15T21:03:59.108938+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1275, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1063, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1273}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12741, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.348561+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:15:59.348561+0000', 'last_peered': '2024-09-15T21:15:59.348561+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:15:59.348561+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12448, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.350830+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:15:59.350830+0000', 'last_peered': '2024-09-15T21:15:59.350830+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:15:59.350830+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:08.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:08.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12856, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.349773+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:59.349773+0000', 'last_peered': '2024-09-15T21:15:59.349773+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:59.349773+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:08.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:08.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12536, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.350421+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:15:59.350421+0000', 'last_peered': '2024-09-15T21:15:59.350421+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:15:59.350421+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:08.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:08.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12669, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.354834+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:15:59.354834+0000', 'last_peered': '2024-09-15T21:15:59.354834+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:15:59.354834+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:08.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:08.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12493, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.351047+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:15:59.351047+0000', 'last_peered': '2024-09-15T21:15:59.351047+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:15:59.351047+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:08.454 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:08.454 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 12514, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.348646+0000', 'last_change': '2024-09-15T21:03:59.672561+0000', 'last_active': '2024-09-15T21:15:59.348646+0000', 'last_peered': '2024-09-15T21:15:59.348646+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:15:59.348646+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2281, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2277}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:08.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:08.846 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:08.846 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 19849/64 objects degraded (31014.062%), 9 pgs degraded, 9 pgs undersized","count":18},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":834,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":108640,"bytes_used":2782093312,"bytes_avail":383764963328,"bytes_total":386547056640,"degraded_objects":19849,"degraded_total":64,"degraded_ratio":310.140625},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:08.860 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:08.860 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:09.174 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:09.174 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:09.189 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2024-09-15T21:16:08.221389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716888,"kb_used_data":1540,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770472,"statfs":{"total":386547056640,"available":383764963328,"internally_reserved":0,"allocated":1576960,"data_stored":717852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12677,"num_write_kb":-50717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-983,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.876582"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16987,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.350954+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:15:59.350954+0000","last_peered":"2024-09-15T21:15:59.350954+0000","last_clean":"2024-09-15T21:15:59.350954+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:15:59.350954+0000","last_undegraded":"2024-09-15T21:15:59.350954+0000","last_fullsized":"2024-09-15T21:15:59.350954+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17465,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349619+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:15:59.349619+0000","last_peered":"2024-09-15T21:15:59.349619+0000","last_clean":"2024-09-15T21:15:59.349619+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:15:59.349619+0000","last_undegraded":"2024-09-15T21:15:59.349619+0000","last_fullsized":"2024-09-15T21:15:59.349619+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12539,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.353044+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:15:59.353044+0000","last_peered":"2024-09-15T21:15:59.353044+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:15:59.353044+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":14674,"reported_epoch":45,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:04.173844+0000","last_change":"2024-09-15T21:15:57.824583+0000","last_active":"2024-09-15T21:16:04.173844+0000","last_peered":"2024-09-15T21:16:04.173844+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:04.173844+0000","last_undegraded":"2024-09-15T21:03:59.108938+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1275,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1063,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1273}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12741,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348561+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:15:59.348561+0000","last_peered":"2024-09-15T21:15:59.348561+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:15:59.348561+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12448,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350830+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:15:59.350830+0000","last_peered":"2024-09-15T21:15:59.350830+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:15:59.350830+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12536,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.350421+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:15:59.350421+0000","last_peered":"2024-09-15T21:15:59.350421+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:15:59.350421+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12669,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.354834+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:15:59.354834+0000","last_peered":"2024-09-15T21:15:59.354834+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:15:59.354834+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.348247+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:15:59.348247+0000","last_peered":"2024-09-15T21:15:59.348247+0000","last_clean":"2024-09-15T21:15:59.348247+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:15:59.348247+0000","last_undegraded":"2024-09-15T21:15:59.348247+0000","last_fullsized":"2024-09-15T21:15:59.348247+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12493,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.351047+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:15:59.351047+0000","last_peered":"2024-09-15T21:15:59.351047+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:15:59.351047+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17210,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.351442+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:15:59.351442+0000","last_peered":"2024-09-15T21:15:59.351442+0000","last_clean":"2024-09-15T21:15:59.351442+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:15:59.351442+0000","last_undegraded":"2024-09-15T21:15:59.351442+0000","last_fullsized":"2024-09-15T21:15:59.351442+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12514,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.348646+0000","last_change":"2024-09-15T21:03:59.672561+0000","last_active":"2024-09-15T21:15:59.348646+0000","last_peered":"2024-09-15T21:15:59.348646+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:15:59.348646+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2281,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2277}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12796,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.721933+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:15:59.721933+0000","last_peered":"2024-09-15T21:15:59.721933+0000","last_clean":"2024-09-15T21:15:59.721933+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:15:59.721933+0000","last_undegraded":"2024-09-15T21:15:59.721933+0000","last_fullsized":"2024-09-15T21:15:59.721933+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13779,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349234+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:15:59.349234+0000","last_peered":"2024-09-15T21:15:59.349234+0000","last_clean":"2024-09-15T21:15:59.349234+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:15:59.349234+0000","last_undegraded":"2024-09-15T21:15:59.349234+0000","last_fullsized":"2024-09-15T21:15:59.349234+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3352,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":69632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708168,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663672,"statfs":{"total":96636764160,"available":95911600128,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695180,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676660,"statfs":{"total":96636764160,"available":95924899840,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051729,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599864,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771976,"statfs":{"total":96636764160,"available":96022503424,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247967,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":57344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:12.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:12.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:12.513 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:12.526 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-15T21:16:12.222111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708592,"kb_used_data":1564,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778768,"statfs":{"total":386547056640,"available":383773458432,"internally_reserved":0,"allocated":1601536,"data_stored":729603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001390"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17467,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384265+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:10.384265+0000","last_peered":"2024-09-15T21:16:10.384265+0000","last_clean":"2024-09-15T21:16:10.384265+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:10.384265+0000","last_undegraded":"2024-09-15T21:16:10.384265+0000","last_fullsized":"2024-09-15T21:16:10.384265+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-15T21:16:09.984678+0000","last_change":"2024-09-15T21:16:10.583613+0000","last_active":"2024-09-15T21:16:10.583613+0000","last_peered":"2024-09-15T21:16:09.984678+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:09.984678+0000","last_undegraded":"2024-09-15T21:16:09.984678+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17212,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384182+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:10.384182+0000","last_peered":"2024-09-15T21:16:10.384182+0000","last_clean":"2024-09-15T21:16:10.384182+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:10.384182+0000","last_undegraded":"2024-09-15T21:16:10.384182+0000","last_fullsized":"2024-09-15T21:16:10.384182+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12798,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384067+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:10.384067+0000","last_peered":"2024-09-15T21:16:10.384067+0000","last_clean":"2024-09-15T21:16:10.384067+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:10.384067+0000","last_undegraded":"2024-09-15T21:16:10.384067+0000","last_fullsized":"2024-09-15T21:16:10.384067+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384113+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:10.384113+0000","last_peered":"2024-09-15T21:16:10.384113+0000","last_clean":"2024-09-15T21:16:10.384113+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:10.384113+0000","last_undegraded":"2024-09-15T21:16:10.384113+0000","last_fullsized":"2024-09-15T21:16:10.384113+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591544,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780296,"statfs":{"total":96636764160,"available":96031023104,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:12.527 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:12.869 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:12.869 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:12.882 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-15T21:16:12.222111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708592,"kb_used_data":1564,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778768,"statfs":{"total":386547056640,"available":383773458432,"internally_reserved":0,"allocated":1601536,"data_stored":729603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001390"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17467,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384265+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:10.384265+0000","last_peered":"2024-09-15T21:16:10.384265+0000","last_clean":"2024-09-15T21:16:10.384265+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:10.384265+0000","last_undegraded":"2024-09-15T21:16:10.384265+0000","last_fullsized":"2024-09-15T21:16:10.384265+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-15T21:16:09.984678+0000","last_change":"2024-09-15T21:16:10.583613+0000","last_active":"2024-09-15T21:16:10.583613+0000","last_peered":"2024-09-15T21:16:09.984678+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:09.984678+0000","last_undegraded":"2024-09-15T21:16:09.984678+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17212,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384182+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:10.384182+0000","last_peered":"2024-09-15T21:16:10.384182+0000","last_clean":"2024-09-15T21:16:10.384182+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:10.384182+0000","last_undegraded":"2024-09-15T21:16:10.384182+0000","last_fullsized":"2024-09-15T21:16:10.384182+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12798,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384067+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:10.384067+0000","last_peered":"2024-09-15T21:16:10.384067+0000","last_clean":"2024-09-15T21:16:10.384067+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:10.384067+0000","last_undegraded":"2024-09-15T21:16:10.384067+0000","last_fullsized":"2024-09-15T21:16:10.384067+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384113+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:10.384113+0000","last_peered":"2024-09-15T21:16:10.384113+0000","last_clean":"2024-09-15T21:16:10.384113+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:10.384113+0000","last_undegraded":"2024-09-15T21:16:10.384113+0000","last_fullsized":"2024-09-15T21:16:10.384113+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591544,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780296,"statfs":{"total":96636764160,"available":96031023104,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:12.883 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:12.883 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12543, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391524+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:11.391524+0000', 'last_peered': '2024-09-15T21:16:11.391524+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:11.391524+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.0 is not active+clean 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.0', 'version': "37'12454", 'reported_seq': 12539, 'reported_epoch': 46, 'state': 'peering', 'last_fresh': '2024-09-15T21:16:09.984678+0000', 'last_change': '2024-09-15T21:16:10.583613+0000', 'last_active': '2024-09-15T21:16:10.583613+0000', 'last_peered': '2024-09-15T21:16:09.984678+0000', 'last_clean': '2024-09-15T21:02:50.161184+0000', 'last_became_active': '2024-09-15T21:03:59.702984+0000', 'last_became_peered': '2024-09-15T21:03:59.702984+0000', 'last_unstale': '2024-09-15T21:16:09.984678+0000', 'last_undegraded': '2024-09-15T21:16:09.984678+0000', 'last_fullsized': '2024-09-15T21:03:59.047696+0000', 'mapping_epoch': 46, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10054, 'log_dups_size': 0, 'ondisk_log_size': 10054, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12452, 'num_write_kb': 49813, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2338, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12745, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388055+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:11.388055+0000', 'last_peered': '2024-09-15T21:16:11.388055+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:11.388055+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12452, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388237+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:11.388237+0000', 'last_peered': '2024-09-15T21:16:11.388237+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:11.388237+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12856, 'reported_epoch': 45, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:15:59.349773+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:15:59.349773+0000', 'last_peered': '2024-09-15T21:15:59.349773+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:15:59.349773+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:12.884 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12540, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.387716+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:11.387716+0000', 'last_peered': '2024-09-15T21:16:11.387716+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:11.387716+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12673, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391786+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:11.391786+0000', 'last_peered': '2024-09-15T21:16:11.391786+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:11.391786+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12497, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.389472+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:11.389472+0000', 'last_peered': '2024-09-15T21:16:11.389472+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:11.389472+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:12.885 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 13069, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.616484+0000', 'last_change': '2024-09-15T21:16:09.985357+0000', 'last_active': '2024-09-15T21:16:11.616484+0000', 'last_peered': '2024-09-15T21:16:11.616484+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:16:11.616484+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 277, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:12.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:13.271 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:13.271 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18299/64 objects degraded (28592.188%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+clean","count":7},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":16,"data_bytes":108640,"bytes_used":2773598208,"bytes_avail":383773458432,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":18299,"degraded_total":64,"degraded_ratio":285.921875,"recovering_objects_per_sec":193,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1550,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:13.284 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:16:13.285 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:13.604 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:13.605 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:13.617 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-15T21:16:12.222111+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708592,"kb_used_data":1564,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778768,"statfs":{"total":386547056640,"available":383773458432,"internally_reserved":0,"allocated":1601536,"data_stored":729603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1550,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001390"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17467,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384265+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:10.384265+0000","last_peered":"2024-09-15T21:16:10.384265+0000","last_clean":"2024-09-15T21:16:10.384265+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:10.384265+0000","last_undegraded":"2024-09-15T21:16:10.384265+0000","last_fullsized":"2024-09-15T21:16:10.384265+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12539,"reported_epoch":46,"state":"peering","last_fresh":"2024-09-15T21:16:09.984678+0000","last_change":"2024-09-15T21:16:10.583613+0000","last_active":"2024-09-15T21:16:10.583613+0000","last_peered":"2024-09-15T21:16:09.984678+0000","last_clean":"2024-09-15T21:02:50.161184+0000","last_became_active":"2024-09-15T21:03:59.702984+0000","last_became_peered":"2024-09-15T21:03:59.702984+0000","last_unstale":"2024-09-15T21:16:09.984678+0000","last_undegraded":"2024-09-15T21:16:09.984678+0000","last_fullsized":"2024-09-15T21:03:59.047696+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12452,"num_write_kb":49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2338,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17622,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-15T21:15:59.349629+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:15:59.349629+0000","last_peered":"2024-09-15T21:15:59.349629+0000","last_clean":"2024-09-15T21:15:59.349629+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:15:59.349629+0000","last_undegraded":"2024-09-15T21:15:59.349629+0000","last_fullsized":"2024-09-15T21:15:59.349629+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12856,"reported_epoch":45,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:15:59.349773+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:15:59.349773+0000","last_peered":"2024-09-15T21:15:59.349773+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:15:59.349773+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17212,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384182+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:10.384182+0000","last_peered":"2024-09-15T21:16:10.384182+0000","last_clean":"2024-09-15T21:16:10.384182+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:10.384182+0000","last_undegraded":"2024-09-15T21:16:10.384182+0000","last_fullsized":"2024-09-15T21:16:10.384182+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12798,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384067+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:10.384067+0000","last_peered":"2024-09-15T21:16:10.384067+0000","last_clean":"2024-09-15T21:16:10.384067+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:10.384067+0000","last_undegraded":"2024-09-15T21:16:10.384067+0000","last_fullsized":"2024-09-15T21:16:10.384067+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13781,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-15T21:16:10.384113+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:10.384113+0000","last_peered":"2024-09-15T21:16:10.384113+0000","last_clean":"2024-09-15T21:16:10.384113+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:10.384113+0000","last_undegraded":"2024-09-15T21:16:10.384113+0000","last_fullsized":"2024-09-15T21:16:10.384113+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":108640,"num_objects":16,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135328,"num_write_kb":501927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051730,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591544,"kb_used_data":424,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780296,"statfs":{"total":96636764160,"available":96031023104,"internally_reserved":0,"allocated":434176,"data_stored":219399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247968,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713676,"kb_used_data":372,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658164,"statfs":{"total":96636764160,"available":95905959936,"internally_reserved":0,"allocated":380928,"data_stored":166151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:16.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:16.925 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:16.925 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:16.937 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-15T21:16:16.222821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716936,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770424,"statfs":{"total":386547056640,"available":383764914176,"internally_reserved":0,"allocated":1626112,"data_stored":737065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-788,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001431"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17214,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388051+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388051+0000","last_peered":"2024-09-15T21:16:11.388051+0000","last_clean":"2024-09-15T21:16:11.388051+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388051+0000","last_undegraded":"2024-09-15T21:16:11.388051+0000","last_fullsized":"2024-09-15T21:16:11.388051+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599872,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771968,"statfs":{"total":96636764160,"available":96022495232,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:16.938 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:17.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:17.258 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:17.271 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-15T21:16:16.222821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716936,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770424,"statfs":{"total":386547056640,"available":383764914176,"internally_reserved":0,"allocated":1626112,"data_stored":737065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-788,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001431"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17214,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388051+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388051+0000","last_peered":"2024-09-15T21:16:11.388051+0000","last_clean":"2024-09-15T21:16:11.388051+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388051+0000","last_undegraded":"2024-09-15T21:16:11.388051+0000","last_fullsized":"2024-09-15T21:16:11.388051+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599872,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771968,"statfs":{"total":96636764160,"available":96022495232,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:17.272 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:17.272 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12543, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391524+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:11.391524+0000', 'last_peered': '2024-09-15T21:16:11.391524+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:11.391524+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12745, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388055+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:11.388055+0000', 'last_peered': '2024-09-15T21:16:11.388055+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:11.388055+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12452, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388237+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:11.388237+0000', 'last_peered': '2024-09-15T21:16:11.388237+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:11.388237+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12860, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.389952+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:16:11.389952+0000', 'last_peered': '2024-09-15T21:16:11.389952+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:11.389952+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12540, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.387716+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:11.387716+0000', 'last_peered': '2024-09-15T21:16:11.387716+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:11.387716+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:17.273 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12673, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391786+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:11.391786+0000', 'last_peered': '2024-09-15T21:16:11.391786+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:11.391786+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:17.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:17.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12497, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.389472+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:11.389472+0000', 'last_peered': '2024-09-15T21:16:11.389472+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:11.389472+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:17.274 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:17.274 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 13069, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.616484+0000', 'last_change': '2024-09-15T21:16:09.985357+0000', 'last_active': '2024-09-15T21:16:11.616484+0000', 'last_peered': '2024-09-15T21:16:11.616484+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:16:11.616484+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2006, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 277, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2004}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:17.274 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:17.673 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:17.674 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18299/56 objects degraded (32676.786%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":95060,"bytes_used":2782142464,"bytes_avail":383764914176,"bytes_total":386547056640,"degraded_objects":18299,"degraded_total":56,"degraded_ratio":326.76785714285717},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:17.687 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:17.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:18.004 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:18.004 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:18.016 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-15T21:16:16.222821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716936,"kb_used_data":1588,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770424,"statfs":{"total":386547056640,"available":383764914176,"internally_reserved":0,"allocated":1626112,"data_stored":737065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1550,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-788,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001431"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12543,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391524+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.391524+0000","last_peered":"2024-09-15T21:16:11.391524+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.391524+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17214,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388051+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388051+0000","last_peered":"2024-09-15T21:16:11.388051+0000","last_clean":"2024-09-15T21:16:11.388051+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388051+0000","last_undegraded":"2024-09-15T21:16:11.388051+0000","last_fullsized":"2024-09-15T21:16:11.388051+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":13069,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.616484+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:11.616484+0000","last_peered":"2024-09-15T21:16:11.616484+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:11.616484+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2006,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2004}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708176,"kb_used_data":380,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663664,"statfs":{"total":96636764160,"available":95911591936,"internally_reserved":0,"allocated":389120,"data_stored":170075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599872,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771968,"statfs":{"total":96636764160,"available":96022495232,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247969,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:21.019 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:21.337 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:21.338 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:21.350 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-15T21:16:20.223598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716944,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770416,"statfs":{"total":386547056640,"available":383764905984,"internally_reserved":0,"allocated":1634304,"data_stored":740968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.842208"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17214,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388051+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388051+0000","last_peered":"2024-09-15T21:16:11.388051+0000","last_clean":"2024-09-15T21:16:11.388051+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388051+0000","last_undegraded":"2024-09-15T21:16:11.388051+0000","last_fullsized":"2024-09-15T21:16:11.388051+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":14699,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:16.618127+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:16.618127+0000","last_peered":"2024-09-15T21:16:16.618127+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:16.618127+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1092,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1189}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599872,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771968,"statfs":{"total":96636764160,"available":96022495232,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:21.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:21.677 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:21.678 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:21.690 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-15T21:16:20.223598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716944,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770416,"statfs":{"total":386547056640,"available":383764905984,"internally_reserved":0,"allocated":1634304,"data_stored":740968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.842208"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12497,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389472+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.389472+0000","last_peered":"2024-09-15T21:16:11.389472+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.389472+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17214,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388051+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388051+0000","last_peered":"2024-09-15T21:16:11.388051+0000","last_clean":"2024-09-15T21:16:11.388051+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388051+0000","last_undegraded":"2024-09-15T21:16:11.388051+0000","last_fullsized":"2024-09-15T21:16:11.388051+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":14699,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:16.618127+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:16.618127+0000","last_peered":"2024-09-15T21:16:16.618127+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:16.618127+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1092,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1189}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17484,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3379,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051731,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599872,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771968,"statfs":{"total":96636764160,"available":96022495232,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:21.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:21.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12544, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.392855+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:11.392855+0000', 'last_peered': '2024-09-15T21:16:11.392855+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:11.392855+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12745, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388055+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:11.388055+0000', 'last_peered': '2024-09-15T21:16:11.388055+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:11.388055+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12452, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388237+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:11.388237+0000', 'last_peered': '2024-09-15T21:16:11.388237+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:11.388237+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12860, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.389952+0000', 'last_change': '2024-09-15T21:03:59.744218+0000', 'last_active': '2024-09-15T21:16:11.389952+0000', 'last_peered': '2024-09-15T21:16:11.389952+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:11.389952+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2476, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2472}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12540, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.387716+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:11.387716+0000', 'last_peered': '2024-09-15T21:16:11.387716+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:11.387716+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:21.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12673, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391786+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:11.391786+0000', 'last_peered': '2024-09-15T21:16:11.391786+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:11.391786+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:21.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:21.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12497, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.389472+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:11.389472+0000', 'last_peered': '2024-09-15T21:16:11.389472+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:11.389472+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:21.693 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:21.693 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 14699, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:16.618127+0000', 'last_change': '2024-09-15T21:16:09.985357+0000', 'last_active': '2024-09-15T21:16:16.618127+0000', 'last_peered': '2024-09-15T21:16:16.618127+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:16:16.618127+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1191, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1092, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1189}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:21.693 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:22.097 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:22.097 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 17484/56 objects degraded (31221.429%), 8 pgs degraded, 8 pgs undersized","count":16},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":847,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":8},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":14,"data_bytes":95060,"bytes_used":2782150656,"bytes_avail":383764905984,"bytes_total":386547056640,"degraded_objects":17484,"degraded_total":56,"degraded_ratio":312.21428571428572},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:22.111 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:22.112 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:22.409 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:22.409 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:22.423 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-15T21:16:22.223923+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708624,"kb_used_data":1596,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778736,"statfs":{"total":386547056640,"available":383773425664,"internally_reserved":0,"allocated":1634304,"data_stored":740968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12452,"num_write_kb":-49813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.837219"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17626,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.391166+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:11.391166+0000","last_peered":"2024-09-15T21:16:11.391166+0000","last_clean":"2024-09-15T21:16:11.391166+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:11.391166+0000","last_undegraded":"2024-09-15T21:16:11.391166+0000","last_fullsized":"2024-09-15T21:16:11.391166+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12860,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.389952+0000","last_change":"2024-09-15T21:03:59.744218+0000","last_active":"2024-09-15T21:16:11.389952+0000","last_peered":"2024-09-15T21:16:11.389952+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:11.389952+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2476,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2472}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12498,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.390686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.390686+0000","last_peered":"2024-09-15T21:16:11.390686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.390686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17215,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388096+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388096+0000","last_peered":"2024-09-15T21:16:11.388096+0000","last_clean":"2024-09-15T21:16:11.388096+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388096+0000","last_undegraded":"2024-09-15T21:16:11.388096+0000","last_fullsized":"2024-09-15T21:16:11.388096+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":16973,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:21.618411+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:21.618411+0000","last_peered":"2024-09-15T21:16:21.618411+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:21.618411+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4516,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591552,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780288,"statfs":{"total":96636764160,"available":96031014912,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247970,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713692,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658148,"statfs":{"total":96636764160,"available":95905943552,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:25.424 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:25.753 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:25.754 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:25.767 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-15T21:16:24.224381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4935,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708628,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778732,"statfs":{"total":386547056640,"available":383773421568,"internally_reserved":0,"allocated":1642496,"data_stored":744855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001559"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":13703,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:24.174895+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:24.174895+0000","last_peered":"2024-09-15T21:16:24.174895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:24.174895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12498,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.390686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.390686+0000","last_peered":"2024-09-15T21:16:11.390686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.390686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17215,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388096+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388096+0000","last_peered":"2024-09-15T21:16:11.388096+0000","last_clean":"2024-09-15T21:16:11.388096+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388096+0000","last_undegraded":"2024-09-15T21:16:11.388096+0000","last_fullsized":"2024-09-15T21:16:11.388096+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":16973,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:21.618411+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:21.618411+0000","last_peered":"2024-09-15T21:16:21.618411+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:21.618411+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4935,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591552,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780288,"statfs":{"total":96636764160,"available":96031014912,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":177865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:25.769 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:26.082 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:26.083 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:26.095 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-15T21:16:24.224381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4935,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2708628,"kb_used_data":1604,"kb_used_omap":14,"kb_used_meta":2706929,"kb_avail":374778732,"statfs":{"total":386547056640,"available":383773421568,"internally_reserved":0,"allocated":1642496,"data_stored":744855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2771895858},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2371,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001559"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17469,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387684+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:11.387684+0000","last_peered":"2024-09-15T21:16:11.387684+0000","last_clean":"2024-09-15T21:16:11.387684+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:11.387684+0000","last_undegraded":"2024-09-15T21:16:11.387684+0000","last_fullsized":"2024-09-15T21:16:11.387684+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12558,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.762034+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:11.762034+0000","last_peered":"2024-09-15T21:16:11.762034+0000","last_clean":"2024-09-15T21:16:11.762034+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:11.762034+0000","last_undegraded":"2024-09-15T21:16:11.762034+0000","last_fullsized":"2024-09-15T21:16:11.762034+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":13703,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:24.174895+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:24.174895+0000","last_peered":"2024-09-15T21:16:24.174895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:24.174895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12498,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.390686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.390686+0000","last_peered":"2024-09-15T21:16:11.390686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.390686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17215,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.388096+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:11.388096+0000","last_peered":"2024-09-15T21:16:11.388096+0000","last_clean":"2024-09-15T21:16:11.388096+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:11.388096+0000","last_undegraded":"2024-09-15T21:16:11.388096+0000","last_fullsized":"2024-09-15T21:16:11.388096+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":16973,"reported_epoch":47,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:21.618411+0000","last_change":"2024-09-15T21:16:09.985357+0000","last_active":"2024-09-15T21:16:21.618411+0000","last_peered":"2024-09-15T21:16:21.618411+0000","last_clean":"2024-09-15T21:02:50.058145+0000","last_became_active":"2024-09-15T21:03:59.666431+0000","last_became_peered":"2024-09-15T21:03:59.666431+0000","last_unstale":"2024-09-15T21:16:21.618411+0000","last_undegraded":"2024-09-15T21:03:59.119237+0000","last_fullsized":"2024-09-15T21:03:59.042718+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":54,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12428,"num_write_kb":49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2229,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-52}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12800,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387934+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:11.387934+0000","last_peered":"2024-09-15T21:16:11.387934+0000","last_clean":"2024-09-15T21:16:11.387934+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:11.387934+0000","last_undegraded":"2024-09-15T21:16:11.387934+0000","last_fullsized":"2024-09-15T21:16:11.387934+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13783,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.387593+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:11.387593+0000","last_peered":"2024-09-15T21:16:11.387593+0000","last_clean":"2024-09-15T21:16:11.387593+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:11.387593+0000","last_undegraded":"2024-09-15T21:16:11.387593+0000","last_fullsized":"2024-09-15T21:16:11.387593+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":95060,"num_objects":14,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15928,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122876,"num_write_kb":452114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4935,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":61440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051732,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":591552,"kb_used_data":432,"kb_used_omap":4,"kb_used_meta":591099,"kb_avail":93780288,"statfs":{"total":96636764160,"available":96031014912,"internally_reserved":0,"allocated":442368,"data_stored":219034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":605286095},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":177865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":49152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:26.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:26.096 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12544, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.392855+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:11.392855+0000', 'last_peered': '2024-09-15T21:16:11.392855+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:11.392855+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:26.096 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12745, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388055+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:11.388055+0000', 'last_peered': '2024-09-15T21:16:11.388055+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:11.388055+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12452, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.388237+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:11.388237+0000', 'last_peered': '2024-09-15T21:16:11.388237+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:11.388237+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 13703, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:24.174895+0000', 'last_change': '2024-09-15T21:16:21.868348+0000', 'last_active': '2024-09-15T21:16:24.174895+0000', 'last_peered': '2024-09-15T21:16:24.174895+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:24.174895+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2057, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 421, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -2055}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12540, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.387716+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:11.387716+0000', 'last_peered': '2024-09-15T21:16:11.387716+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:11.387716+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12673, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.391786+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:11.391786+0000', 'last_peered': '2024-09-15T21:16:11.391786+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:11.391786+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:26.097 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12498, 'reported_epoch': 47, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:11.390686+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:11.390686+0000', 'last_peered': '2024-09-15T21:16:11.390686+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:11.390686+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:26.098 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.9 is not active+clean 2024-09-15T21:16:26.098 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.9', 'version': "37'12430", 'reported_seq': 16973, 'reported_epoch': 47, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:21.618411+0000', 'last_change': '2024-09-15T21:16:09.985357+0000', 'last_active': '2024-09-15T21:16:21.618411+0000', 'last_peered': '2024-09-15T21:16:21.618411+0000', 'last_clean': '2024-09-15T21:02:50.058145+0000', 'last_became_active': '2024-09-15T21:03:59.666431+0000', 'last_became_peered': '2024-09-15T21:03:59.666431+0000', 'last_unstale': '2024-09-15T21:16:21.618411+0000', 'last_undegraded': '2024-09-15T21:03:59.119237+0000', 'last_fullsized': '2024-09-15T21:03:59.042718+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10030, 'log_dups_size': 0, 'ondisk_log_size': 10030, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 54, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12428, 'num_write_kb': 49724, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2229, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -52}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:26.098 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:26.479 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:26.480 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15874/48 objects degraded (33070.833%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":852,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":81480,"bytes_used":2782162944,"bytes_avail":383764893696,"bytes_total":386547056640,"degraded_objects":15874,"degraded_total":48,"degraded_ratio":330.70833333333326},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:26.493 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:26.493 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:26.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:26.793 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:26.805 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":466,"stamp":"2024-09-15T21:16:26.224725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2706,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716956,"kb_used_data":1612,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770404,"statfs":{"total":386547056640,"available":383764893696,"internally_reserved":0,"allocated":1650688,"data_stored":744416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001564"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16991,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389386+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:11.389386+0000","last_peered":"2024-09-15T21:16:11.389386+0000","last_clean":"2024-09-15T21:16:11.389386+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:11.389386+0000","last_undegraded":"2024-09-15T21:16:11.389386+0000","last_fullsized":"2024-09-15T21:16:11.389386+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12544,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.392855+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:11.392855+0000","last_peered":"2024-09-15T21:16:11.392855+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:11.392855+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12745,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388055+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:11.388055+0000","last_peered":"2024-09-15T21:16:11.388055+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:11.388055+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12452,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.388237+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:11.388237+0000","last_peered":"2024-09-15T21:16:11.388237+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:11.388237+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":13703,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:24.174895+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:24.174895+0000","last_peered":"2024-09-15T21:16:24.174895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:24.174895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12540,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.387716+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:11.387716+0000","last_peered":"2024-09-15T21:16:11.387716+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:11.387716+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12673,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.391786+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:11.391786+0000","last_peered":"2024-09-15T21:16:11.391786+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:11.391786+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17055,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-15T21:16:11.389223+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:11.389223+0000","last_peered":"2024-09-15T21:16:11.389223+0000","last_clean":"2024-09-15T21:16:11.389223+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:11.389223+0000","last_undegraded":"2024-09-15T21:16:11.389223+0000","last_fullsized":"2024-09-15T21:16:11.389223+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12498,"reported_epoch":47,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:11.390686+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:11.390686+0000","last_peered":"2024-09-15T21:16:11.390686+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:11.390686+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2706,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708184,"kb_used_data":388,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663656,"statfs":{"total":96636764160,"available":95911583744,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695196,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676644,"statfs":{"total":96636764160,"available":95924883456,"internally_reserved":0,"allocated":397312,"data_stored":173978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599880,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771960,"statfs":{"total":96636764160,"available":96022487040,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":177865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:29.807 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:30.154 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:30.154 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:30.167 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-15T21:16:28.225072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2706,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2716988,"kb_used_data":1644,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770372,"statfs":{"total":386547056640,"available":383764860928,"internally_reserved":0,"allocated":1683456,"data_stored":759922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1610,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-673,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001473"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":13703,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:24.174895+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:24.174895+0000","last_peered":"2024-09-15T21:16:24.174895+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:24.174895+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2057,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":421,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-2055}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15874,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2706,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599880,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771960,"statfs":{"total":96636764160,"available":96022487040,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247971,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713696,"kb_used_data":396,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658144,"statfs":{"total":96636764160,"available":95905939456,"internally_reserved":0,"allocated":405504,"data_stored":177865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:30.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:30.499 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:30.500 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:30.512 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-15T21:16:30.225402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717000,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770360,"statfs":{"total":386547056640,"available":383764848640,"internally_reserved":0,"allocated":1691648,"data_stored":763788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-998,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001479"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":15327,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:29.179777+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:29.179777+0000","last_peered":"2024-09-15T21:16:29.179777+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:29.179777+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599880,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771960,"statfs":{"total":96636764160,"available":96022487040,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:30.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:30.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12548, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.429343+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:23.429343+0000', 'last_peered': '2024-09-15T21:16:23.429343+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:23.429343+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:30.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:30.513 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12749, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.425409+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:23.425409+0000', 'last_peered': '2024-09-15T21:16:23.425409+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:23.425409+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:30.513 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12456, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.426696+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:23.426696+0000', 'last_peered': '2024-09-15T21:16:23.426696+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:23.426696+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 15327, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:29.179777+0000', 'last_change': '2024-09-15T21:16:21.868348+0000', 'last_active': '2024-09-15T21:16:29.179777+0000', 'last_peered': '2024-09-15T21:16:29.179777+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:29.179777+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1245, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1233, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -1243}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12544, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.425914+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:23.425914+0000', 'last_peered': '2024-09-15T21:16:23.425914+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:23.425914+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12677, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.429551+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:23.429551+0000', 'last_peered': '2024-09-15T21:16:23.429551+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:23.429551+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:30.514 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12502, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.426507+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:23.426507+0000', 'last_peered': '2024-09-15T21:16:23.426507+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:23.426507+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:30.515 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:30.934 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:30.934 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 15062/48 objects degraded (31379.167%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":856,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":81480,"bytes_used":2782208000,"bytes_avail":383764848640,"bytes_total":386547056640,"degraded_objects":15062,"degraded_total":48,"degraded_ratio":313.79166666666669},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:30.948 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:30.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:31.296 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:31.296 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:31.309 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":468,"stamp":"2024-09-15T21:16:30.225402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2717000,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2715249,"kb_avail":374770360,"statfs":{"total":386547056640,"available":383764848640,"internally_reserved":0,"allocated":1691648,"data_stored":763788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2780415538},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1285,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-998,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001479"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":15327,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:29.179777+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:29.179777+0000","last_peered":"2024-09-15T21:16:29.179777+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:29.179777+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1233,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-1243}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15062,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3518,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051733,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":599880,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":599419,"kb_avail":93771960,"statfs":{"total":96636764160,"available":96022487040,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":613805775},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247972,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:34.311 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:34.621 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:34.622 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:34.636 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-15T21:16:34.226116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704520,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782840,"statfs":{"total":386547056640,"available":383777628160,"internally_reserved":0,"allocated":1691648,"data_stored":763788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.803257"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":17459,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:34.181847+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:34.181847+0000","last_peered":"2024-09-15T21:16:34.181847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.181847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587400,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784440,"statfs":{"total":96636764160,"available":96035266560,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:34.637 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:34.948 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:34.949 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:34.962 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-15T21:16:34.226116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704520,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782840,"statfs":{"total":386547056640,"available":383777628160,"internally_reserved":0,"allocated":1691648,"data_stored":763788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2351,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.803257"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":17459,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:34.181847+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:34.181847+0000","last_peered":"2024-09-15T21:16:34.181847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.181847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587400,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784440,"statfs":{"total":96636764160,"available":96035266560,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12548, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.429343+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:23.429343+0000', 'last_peered': '2024-09-15T21:16:23.429343+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:23.429343+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12749, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.425409+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:23.425409+0000', 'last_peered': '2024-09-15T21:16:23.425409+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:23.425409+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:34.963 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12456, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.426696+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:23.426696+0000', 'last_peered': '2024-09-15T21:16:23.426696+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:23.426696+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 17459, 'reported_epoch': 49, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:34.181847+0000', 'last_change': '2024-09-15T21:16:21.868348+0000', 'last_active': '2024-09-15T21:16:34.181847+0000', 'last_peered': '2024-09-15T21:16:34.181847+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:34.181847+0000', 'last_undegraded': '2024-09-15T21:03:59.108199+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 30, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 179, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2299, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [2147483647, 0, 2, 3], 'avail_no_missing': ['0(1)', '2(2)', '3(3)'], 'object_location_counts': [{'shards': '0(1),2(2),3(3)', 'objects': -177}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 0, 'purged_snaps': []} 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12544, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.425914+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:23.425914+0000', 'last_peered': '2024-09-15T21:16:23.425914+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:23.425914+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 12677, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.429551+0000', 'last_change': '2024-09-15T21:03:59.725729+0000', 'last_active': '2024-09-15T21:16:23.429551+0000', 'last_peered': '2024-09-15T21:16:23.429551+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:23.429551+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2337, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -2333}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:34.964 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12502, 'reported_epoch': 49, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:23.426507+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:23.426507+0000', 'last_peered': '2024-09-15T21:16:23.426507+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:23.426507+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:34.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:35.365 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:35.365 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13996/48 objects degraded (29158.333%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":861,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":49,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":81480,"bytes_used":2769428480,"bytes_avail":383777628160,"bytes_total":386547056640,"degraded_objects":13996,"degraded_total":48,"degraded_ratio":291.58333333333331,"recovering_objects_per_sec":6,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":68,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:35.379 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:16:35.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:35.736 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:35.737 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:35.753 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-15T21:16:35.465668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704520,"kb_used_data":1652,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782840,"statfs":{"total":386547056640,"available":383777628160,"internally_reserved":0,"allocated":1691648,"data_stored":763788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1932,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12428,"num_write_kb":-49724,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-351,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.241287"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16995,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425469+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:23.425469+0000","last_peered":"2024-09-15T21:16:23.425469+0000","last_clean":"2024-09-15T21:16:23.425469+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:23.425469+0000","last_undegraded":"2024-09-15T21:16:23.425469+0000","last_fullsized":"2024-09-15T21:16:23.425469+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17473,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.425114+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:23.425114+0000","last_peered":"2024-09-15T21:16:23.425114+0000","last_clean":"2024-09-15T21:16:23.425114+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:23.425114+0000","last_undegraded":"2024-09-15T21:16:23.425114+0000","last_fullsized":"2024-09-15T21:16:23.425114+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12548,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429343+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:23.429343+0000","last_peered":"2024-09-15T21:16:23.429343+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:23.429343+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12562,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424372+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:23.424372+0000","last_peered":"2024-09-15T21:16:23.424372+0000","last_clean":"2024-09-15T21:16:23.424372+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:23.424372+0000","last_undegraded":"2024-09-15T21:16:23.424372+0000","last_fullsized":"2024-09-15T21:16:23.424372+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12749,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425409+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:23.425409+0000","last_peered":"2024-09-15T21:16:23.425409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:23.425409+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12456,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426696+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:23.426696+0000","last_peered":"2024-09-15T21:16:23.426696+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:23.426696+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":17459,"reported_epoch":49,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:34.181847+0000","last_change":"2024-09-15T21:16:21.868348+0000","last_active":"2024-09-15T21:16:34.181847+0000","last_peered":"2024-09-15T21:16:34.181847+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.181847+0000","last_undegraded":"2024-09-15T21:03:59.108199+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":30,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2299,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[2147483647,0,2,3],"avail_no_missing":["0(1)","2(2)","3(3)"],"object_location_counts":[{"shards":"0(1),2(2),3(3)","objects":-177}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12544,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.425914+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:23.425914+0000","last_peered":"2024-09-15T21:16:23.425914+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:23.425914+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":12677,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.429551+0000","last_change":"2024-09-15T21:03:59.725729+0000","last_active":"2024-09-15T21:16:23.429551+0000","last_peered":"2024-09-15T21:16:23.429551+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:23.429551+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2337,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-2333}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17059,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.426296+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:23.426296+0000","last_peered":"2024-09-15T21:16:23.426296+0000","last_clean":"2024-09-15T21:16:23.426296+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:23.426296+0000","last_undegraded":"2024-09-15T21:16:23.426296+0000","last_fullsized":"2024-09-15T21:16:23.426296+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12502,"reported_epoch":49,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:23.426507+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:23.426507+0000","last_peered":"2024-09-15T21:16:23.426507+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:23.426507+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17219,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424477+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:23.424477+0000","last_peered":"2024-09-15T21:16:23.424477+0000","last_clean":"2024-09-15T21:16:23.424477+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:23.424477+0000","last_undegraded":"2024-09-15T21:16:23.424477+0000","last_fullsized":"2024-09-15T21:16:23.424477+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12521,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.610004+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:23.610004+0000","last_peered":"2024-09-15T21:16:23.610004+0000","last_clean":"2024-09-15T21:16:23.610004+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:23.610004+0000","last_undegraded":"2024-09-15T21:16:23.610004+0000","last_fullsized":"2024-09-15T21:16:23.610004+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12804,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.424550+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:23.424550+0000","last_peered":"2024-09-15T21:16:23.424550+0000","last_clean":"2024-09-15T21:16:23.424550+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:23.424550+0000","last_undegraded":"2024-09-15T21:16:23.424550+0000","last_fullsized":"2024-09-15T21:16:23.424550+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13787,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.429175+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:23.429175+0000","last_peered":"2024-09-15T21:16:23.429175+0000","last_clean":"2024-09-15T21:16:23.429175+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:23.429175+0000","last_undegraded":"2024-09-15T21:16:23.429175+0000","last_fullsized":"2024-09-15T21:16:23.429175+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13996,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4584,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":53248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149858,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708200,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663640,"statfs":{"total":96636764160,"available":95911567360,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695212,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676628,"statfs":{"total":96636764160,"available":95924867072,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051734,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587400,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784440,"statfs":{"total":96636764160,"available":96035266560,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:38.755 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:39.058 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:39.058 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:39.071 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-15T21:16:38.226778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712872,"kb_used_data":1684,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774488,"statfs":{"total":386547056640,"available":383769075712,"internally_reserved":0,"allocated":1724416,"data_stored":779146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001376"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17475,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472719+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:35.472719+0000","last_peered":"2024-09-15T21:16:35.472719+0000","last_clean":"2024-09-15T21:16:35.472719+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:35.472719+0000","last_undegraded":"2024-09-15T21:16:35.472719+0000","last_fullsized":"2024-09-15T21:16:35.472719+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12564,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472281+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:35.472281+0000","last_peered":"2024-09-15T21:16:35.472281+0000","last_clean":"2024-09-15T21:16:35.472281+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:35.472281+0000","last_undegraded":"2024-09-15T21:16:35.472281+0000","last_fullsized":"2024-09-15T21:16:35.472281+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-15T21:16:34.933683+0000","last_change":"2024-09-15T21:16:35.590157+0000","last_active":"2024-09-15T21:16:35.590157+0000","last_peered":"2024-09-15T21:16:34.933683+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.933683+0000","last_undegraded":"2024-09-15T21:16:34.933683+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":13442,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:37.107481+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:37.107481+0000","last_peered":"2024-09-15T21:16:37.107481+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:37.107481+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17221,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472412+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:35.472412+0000","last_peered":"2024-09-15T21:16:35.472412+0000","last_clean":"2024-09-15T21:16:35.472412+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:35.472412+0000","last_undegraded":"2024-09-15T21:16:35.472412+0000","last_fullsized":"2024-09-15T21:16:35.472412+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12523,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472505+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:35.472505+0000","last_peered":"2024-09-15T21:16:35.472505+0000","last_clean":"2024-09-15T21:16:35.472505+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:35.472505+0000","last_undegraded":"2024-09-15T21:16:35.472505+0000","last_fullsized":"2024-09-15T21:16:35.472505+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12806,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472569+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:35.472569+0000","last_peered":"2024-09-15T21:16:35.472569+0000","last_clean":"2024-09-15T21:16:35.472569+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:35.472569+0000","last_undegraded":"2024-09-15T21:16:35.472569+0000","last_fullsized":"2024-09-15T21:16:35.472569+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13789,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.473202+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:35.473202+0000","last_peered":"2024-09-15T21:16:35.473202+0000","last_clean":"2024-09-15T21:16:35.473202+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:35.473202+0000","last_undegraded":"2024-09-15T21:16:35.473202+0000","last_fullsized":"2024-09-15T21:16:35.473202+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595720,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776120,"statfs":{"total":96636764160,"available":96026746880,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:39.072 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:39.408 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:39.408 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:39.422 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-15T21:16:38.226778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712872,"kb_used_data":1684,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774488,"statfs":{"total":386547056640,"available":383769075712,"internally_reserved":0,"allocated":1724416,"data_stored":779146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001376"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17475,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472719+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:35.472719+0000","last_peered":"2024-09-15T21:16:35.472719+0000","last_clean":"2024-09-15T21:16:35.472719+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:35.472719+0000","last_undegraded":"2024-09-15T21:16:35.472719+0000","last_fullsized":"2024-09-15T21:16:35.472719+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12564,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472281+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:35.472281+0000","last_peered":"2024-09-15T21:16:35.472281+0000","last_clean":"2024-09-15T21:16:35.472281+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:35.472281+0000","last_undegraded":"2024-09-15T21:16:35.472281+0000","last_fullsized":"2024-09-15T21:16:35.472281+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-15T21:16:34.933683+0000","last_change":"2024-09-15T21:16:35.590157+0000","last_active":"2024-09-15T21:16:35.590157+0000","last_peered":"2024-09-15T21:16:34.933683+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.933683+0000","last_undegraded":"2024-09-15T21:16:34.933683+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":13442,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:37.107481+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:37.107481+0000","last_peered":"2024-09-15T21:16:37.107481+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:37.107481+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17221,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472412+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:35.472412+0000","last_peered":"2024-09-15T21:16:35.472412+0000","last_clean":"2024-09-15T21:16:35.472412+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:35.472412+0000","last_undegraded":"2024-09-15T21:16:35.472412+0000","last_fullsized":"2024-09-15T21:16:35.472412+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12523,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472505+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:35.472505+0000","last_peered":"2024-09-15T21:16:35.472505+0000","last_clean":"2024-09-15T21:16:35.472505+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:35.472505+0000","last_undegraded":"2024-09-15T21:16:35.472505+0000","last_fullsized":"2024-09-15T21:16:35.472505+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12806,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472569+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:35.472569+0000","last_peered":"2024-09-15T21:16:35.472569+0000","last_clean":"2024-09-15T21:16:35.472569+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:35.472569+0000","last_undegraded":"2024-09-15T21:16:35.472569+0000","last_fullsized":"2024-09-15T21:16:35.472569+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13789,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.473202+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:35.473202+0000","last_peered":"2024-09-15T21:16:35.473202+0000","last_clean":"2024-09-15T21:16:35.473202+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:35.473202+0000","last_undegraded":"2024-09-15T21:16:35.473202+0000","last_fullsized":"2024-09-15T21:16:35.473202+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595720,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776120,"statfs":{"total":96636764160,"available":96026746880,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:39.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:39.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12552, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.470375+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:36.470375+0000', 'last_peered': '2024-09-15T21:16:36.470375+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:36.470375+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:39.423 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:39.423 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12753, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.472017+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:36.472017+0000', 'last_peered': '2024-09-15T21:16:36.472017+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:36.472017+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:39.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:39.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12460, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.472491+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:36.472491+0000', 'last_peered': '2024-09-15T21:16:36.472491+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:36.472491+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:39.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.3 is not active+clean 2024-09-15T21:16:39.424 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.3', 'version': "37'12759", 'reported_seq': 12843, 'reported_epoch': 50, 'state': 'peering', 'last_fresh': '2024-09-15T21:16:34.933683+0000', 'last_change': '2024-09-15T21:16:35.590157+0000', 'last_active': '2024-09-15T21:16:35.590157+0000', 'last_peered': '2024-09-15T21:16:34.933683+0000', 'last_clean': '2024-09-15T21:02:50.519912+0000', 'last_became_active': '2024-09-15T21:03:59.704473+0000', 'last_became_peered': '2024-09-15T21:03:59.704473+0000', 'last_unstale': '2024-09-15T21:16:34.933683+0000', 'last_undegraded': '2024-09-15T21:16:34.933683+0000', 'last_fullsized': '2024-09-15T21:03:59.048184+0000', 'mapping_epoch': 50, 'log_start': "31'2700", 'ondisk_log_start': "31'2700", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10059, 'log_dups_size': 0, 'ondisk_log_size': 10059, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12757, 'num_write_kb': 51036, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2478, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0, 2, 3], 'acting': [1, 0, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-15T21:16:39.424 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:39.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12548, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471247+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:36.471247+0000', 'last_peered': '2024-09-15T21:16:36.471247+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:36.471247+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:39.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:39.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 13442, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:37.107481+0000', 'last_change': '2024-09-15T21:16:34.934258+0000', 'last_active': '2024-09-15T21:16:37.107481+0000', 'last_peered': '2024-09-15T21:16:37.107481+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:37.107481+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1957, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 382, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1955}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:39.425 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:39.425 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12506, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471968+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:36.471968+0000', 'last_peered': '2024-09-15T21:16:36.471968+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:36.471968+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:39.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:39.807 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:39.807 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13437/48 objects degraded (27993.750%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":9},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":12,"data_bytes":81480,"bytes_used":2777980928,"bytes_avail":383769075712,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":13437,"degraded_total":48,"degraded_ratio":279.9375,"recovering_objects_per_sec":203,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:39.820 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:16:39.821 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:40.152 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:40.152 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:40.164 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-15T21:16:38.226778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712872,"kb_used_data":1684,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774488,"statfs":{"total":386547056640,"available":383769075712,"internally_reserved":0,"allocated":1724416,"data_stored":779146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1625,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001376"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17475,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472719+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:35.472719+0000","last_peered":"2024-09-15T21:16:35.472719+0000","last_clean":"2024-09-15T21:16:35.472719+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:35.472719+0000","last_undegraded":"2024-09-15T21:16:35.472719+0000","last_fullsized":"2024-09-15T21:16:35.472719+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12564,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472281+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:35.472281+0000","last_peered":"2024-09-15T21:16:35.472281+0000","last_clean":"2024-09-15T21:16:35.472281+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:35.472281+0000","last_undegraded":"2024-09-15T21:16:35.472281+0000","last_fullsized":"2024-09-15T21:16:35.472281+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17630,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-15T21:16:23.428155+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:23.428155+0000","last_peered":"2024-09-15T21:16:23.428155+0000","last_clean":"2024-09-15T21:16:23.428155+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:23.428155+0000","last_undegraded":"2024-09-15T21:16:23.428155+0000","last_fullsized":"2024-09-15T21:16:23.428155+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12843,"reported_epoch":50,"state":"peering","last_fresh":"2024-09-15T21:16:34.933683+0000","last_change":"2024-09-15T21:16:35.590157+0000","last_active":"2024-09-15T21:16:35.590157+0000","last_peered":"2024-09-15T21:16:34.933683+0000","last_clean":"2024-09-15T21:02:50.519912+0000","last_became_active":"2024-09-15T21:03:59.704473+0000","last_became_peered":"2024-09-15T21:03:59.704473+0000","last_unstale":"2024-09-15T21:16:34.933683+0000","last_undegraded":"2024-09-15T21:16:34.933683+0000","last_fullsized":"2024-09-15T21:03:59.048184+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":13442,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:37.107481+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:37.107481+0000","last_peered":"2024-09-15T21:16:37.107481+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:37.107481+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1957,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":382,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1955}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17221,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472412+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:35.472412+0000","last_peered":"2024-09-15T21:16:35.472412+0000","last_clean":"2024-09-15T21:16:35.472412+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:35.472412+0000","last_undegraded":"2024-09-15T21:16:35.472412+0000","last_fullsized":"2024-09-15T21:16:35.472412+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12523,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472505+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:35.472505+0000","last_peered":"2024-09-15T21:16:35.472505+0000","last_clean":"2024-09-15T21:16:35.472505+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:35.472505+0000","last_undegraded":"2024-09-15T21:16:35.472505+0000","last_fullsized":"2024-09-15T21:16:35.472505+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12806,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.472569+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:35.472569+0000","last_peered":"2024-09-15T21:16:35.472569+0000","last_clean":"2024-09-15T21:16:35.472569+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:35.472569+0000","last_undegraded":"2024-09-15T21:16:35.472569+0000","last_fullsized":"2024-09-15T21:16:35.472569+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13789,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-15T21:16:35.473202+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:35.473202+0000","last_peered":"2024-09-15T21:16:35.473202+0000","last_clean":"2024-09-15T21:16:35.473202+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:35.473202+0000","last_undegraded":"2024-09-15T21:16:35.473202+0000","last_fullsized":"2024-09-15T21:16:35.473202+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":81480,"num_objects":12,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13437,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":110448,"num_write_kb":402390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5143,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051735,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595720,"kb_used_data":440,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776120,"statfs":{"total":96636764160,"available":96026746880,"internally_reserved":0,"allocated":450560,"data_stored":218595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247973,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713708,"kb_used_data":404,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658132,"statfs":{"total":96636764160,"available":95905927168,"internally_reserved":0,"allocated":413696,"data_stored":181731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:43.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:43.489 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:43.490 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:43.503 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-15T21:16:42.227413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704572,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782788,"statfs":{"total":386547056640,"available":383777574912,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1060,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":15160,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:42.109889+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:42.109889+0000","last_peered":"2024-09-15T21:16:42.109889+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:42.109889+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587408,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784432,"statfs":{"total":96636764160,"available":96035258368,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:43.504 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:43.822 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:43.822 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:43.834 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-15T21:16:42.227413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704572,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782788,"statfs":{"total":386547056640,"available":383777574912,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1060,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":15160,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:42.109889+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:42.109889+0000","last_peered":"2024-09-15T21:16:42.109889+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:42.109889+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587408,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784432,"statfs":{"total":96636764160,"available":96035258368,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247974,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:43.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:43.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12552, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.470375+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:36.470375+0000', 'last_peered': '2024-09-15T21:16:36.470375+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:36.470375+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:43.835 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:43.835 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12753, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.472017+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:36.472017+0000', 'last_peered': '2024-09-15T21:16:36.472017+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:36.472017+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12460, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.472491+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:36.472491+0000', 'last_peered': '2024-09-15T21:16:36.472491+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:36.472491+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12548, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471247+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:36.471247+0000', 'last_peered': '2024-09-15T21:16:36.471247+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:36.471247+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 15160, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:42.109889+0000', 'last_change': '2024-09-15T21:16:34.934258+0000', 'last_active': '2024-09-15T21:16:42.109889+0000', 'last_peered': '2024-09-15T21:16:42.109889+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:42.109889+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1098, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1241, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -1096}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:43.836 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12506, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471968+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:36.471968+0000', 'last_peered': '2024-09-15T21:16:36.471968+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:36.471968+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:43.837 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:44.263 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:44.263 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 12578/40 objects degraded (31445.000%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":870,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":51,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67900,"bytes_used":2769481728,"bytes_avail":383777574912,"bytes_total":386547056640,"degraded_objects":12578,"degraded_total":40,"degraded_ratio":314.44999999999999},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:44.277 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:44.277 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:44.623 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:44.623 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:44.637 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-09-15T21:16:44.227755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704572,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782788,"statfs":{"total":386547056640,"available":383777574912,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1060,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.762086"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12460,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472491+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.472491+0000","last_peered":"2024-09-15T21:16:36.472491+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.472491+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":15160,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:42.109889+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:42.109889+0000","last_peered":"2024-09-15T21:16:42.109889+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:42.109889+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1098,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1241,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-1096}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3524,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051736,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587408,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784432,"statfs":{"total":96636764160,"available":96035258368,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:47.639 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:47.979 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:47.979 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:47.993 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-09-15T21:16:47.503106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712892,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774468,"statfs":{"total":386547056640,"available":383769055232,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-156,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.034185"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12461,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.473665+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.473665+0000","last_peered":"2024-09-15T21:16:36.473665+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.473665+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17326,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:47.109090+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:47.109090+0000","last_peered":"2024-09-15T21:16:47.109090+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:47.109090+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595728,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776112,"statfs":{"total":96636764160,"available":96026738688,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:47.995 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:48.439 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:48.439 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:48.451 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-15T21:16:48.228454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712892,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774468,"statfs":{"total":386547056640,"available":383769055232,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001676"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12461,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.473665+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.473665+0000","last_peered":"2024-09-15T21:16:36.473665+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.473665+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17326,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:47.109090+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:47.109090+0000","last_peered":"2024-09-15T21:16:47.109090+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:47.109090+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595728,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776112,"statfs":{"total":96636764160,"available":96026738688,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:48.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:48.452 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12552, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.470375+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:36.470375+0000', 'last_peered': '2024-09-15T21:16:36.470375+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:36.470375+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:48.452 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12753, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.472017+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:36.472017+0000', 'last_peered': '2024-09-15T21:16:36.472017+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:36.472017+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12461, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.473665+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:36.473665+0000', 'last_peered': '2024-09-15T21:16:36.473665+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:36.473665+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12548, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471247+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:36.471247+0000', 'last_peered': '2024-09-15T21:16:36.471247+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:36.471247+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.5 is not active+clean 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.5', 'version': "37'12560", 'reported_seq': 17326, 'reported_epoch': 51, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:47.109090+0000', 'last_change': '2024-09-15T21:16:34.934258+0000', 'last_active': '2024-09-15T21:16:47.109090+0000', 'last_peered': '2024-09-15T21:16:47.109090+0000', 'last_clean': '2024-09-15T21:02:42.002893+0000', 'last_became_active': '2024-09-15T21:03:59.708056+0000', 'last_became_peered': '2024-09-15T21:03:59.708056+0000', 'last_unstale': '2024-09-15T21:16:47.109090+0000', 'last_undegraded': '2024-09-15T21:03:59.096585+0000', 'last_fullsized': '2024-09-15T21:03:59.042006+0000', 'mapping_epoch': 30, 'log_start': "31'2500", 'ondisk_log_start': "31'2500", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10060, 'log_dups_size': 0, 'ondisk_log_size': 10060, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 15, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12558, 'num_write_kb': 50238, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2324, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0, 3, 1], 'acting': [2, 0, 3, 2147483647], 'avail_no_missing': ['2(0)', '0(1)', '3(2)'], 'object_location_counts': [{'shards': '0(1),2(0),3(2)', 'objects': -13}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:48.453 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 12506, 'reported_epoch': 51, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:36.471968+0000', 'last_change': '2024-09-15T21:03:59.680936+0000', 'last_active': '2024-09-15T21:16:36.471968+0000', 'last_peered': '2024-09-15T21:16:36.471968+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:36.471968+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2257, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -2255}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:48.454 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:48.915 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:48.915 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 11495/40 objects degraded (28737.500%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":874,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":10},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":10,"data_bytes":67900,"bytes_used":2778001408,"bytes_avail":383769055232,"bytes_total":386547056640,"degraded_objects":11495,"degraded_total":40,"degraded_ratio":287.375},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:48.929 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:48.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:49.497 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:49.498 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:49.510 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-15T21:16:48.510555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712892,"kb_used_data":1708,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774468,"statfs":{"total":386547056640,"available":383769055232,"internally_reserved":0,"allocated":1748992,"data_stored":786312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1942,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12757,"num_write_kb":-51036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.283441"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":16999,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471859+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:36.471859+0000","last_peered":"2024-09-15T21:16:36.471859+0000","last_clean":"2024-09-15T21:16:36.471859+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:36.471859+0000","last_undegraded":"2024-09-15T21:16:36.471859+0000","last_fullsized":"2024-09-15T21:16:36.471859+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17477,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475214+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:36.475214+0000","last_peered":"2024-09-15T21:16:36.475214+0000","last_clean":"2024-09-15T21:16:36.475214+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:36.475214+0000","last_undegraded":"2024-09-15T21:16:36.475214+0000","last_fullsized":"2024-09-15T21:16:36.475214+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12552,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.470375+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:36.470375+0000","last_peered":"2024-09-15T21:16:36.470375+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:36.470375+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12566,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474911+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:36.474911+0000","last_peered":"2024-09-15T21:16:36.474911+0000","last_clean":"2024-09-15T21:16:36.474911+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:36.474911+0000","last_undegraded":"2024-09-15T21:16:36.474911+0000","last_fullsized":"2024-09-15T21:16:36.474911+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12753,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.472017+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:36.472017+0000","last_peered":"2024-09-15T21:16:36.472017+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:36.472017+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12461,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.473665+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:36.473665+0000","last_peered":"2024-09-15T21:16:36.473665+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:36.473665+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17634,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475621+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:36.475621+0000","last_peered":"2024-09-15T21:16:36.475621+0000","last_clean":"2024-09-15T21:16:36.475621+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:36.475621+0000","last_undegraded":"2024-09-15T21:16:36.475621+0000","last_fullsized":"2024-09-15T21:16:36.475621+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12862,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.673452+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:36.673452+0000","last_peered":"2024-09-15T21:16:36.673452+0000","last_clean":"2024-09-15T21:16:36.673452+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:36.673452+0000","last_undegraded":"2024-09-15T21:16:36.673452+0000","last_fullsized":"2024-09-15T21:16:36.673452+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12548,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471247+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:36.471247+0000","last_peered":"2024-09-15T21:16:36.471247+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:36.471247+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17326,"reported_epoch":51,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:47.109090+0000","last_change":"2024-09-15T21:16:34.934258+0000","last_active":"2024-09-15T21:16:47.109090+0000","last_peered":"2024-09-15T21:16:47.109090+0000","last_clean":"2024-09-15T21:02:42.002893+0000","last_became_active":"2024-09-15T21:03:59.708056+0000","last_became_peered":"2024-09-15T21:03:59.708056+0000","last_unstale":"2024-09-15T21:16:47.109090+0000","last_undegraded":"2024-09-15T21:03:59.096585+0000","last_fullsized":"2024-09-15T21:03:59.042006+0000","mapping_epoch":30,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12558,"num_write_kb":50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,2147483647],"avail_no_missing":["2(0)","0(1)","3(2)"],"object_location_counts":[{"shards":"0(1),2(0),3(2)","objects":-13}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17063,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.471680+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:36.471680+0000","last_peered":"2024-09-15T21:16:36.471680+0000","last_clean":"2024-09-15T21:16:36.471680+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:36.471680+0000","last_undegraded":"2024-09-15T21:16:36.471680+0000","last_fullsized":"2024-09-15T21:16:36.471680+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12506,"reported_epoch":51,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:36.471968+0000","last_change":"2024-09-15T21:03:59.680936+0000","last_active":"2024-09-15T21:16:36.471968+0000","last_peered":"2024-09-15T21:16:36.471968+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:36.471968+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2257,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-2255}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17223,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.474945+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:36.474945+0000","last_peered":"2024-09-15T21:16:36.474945+0000","last_clean":"2024-09-15T21:16:36.474945+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:36.474945+0000","last_undegraded":"2024-09-15T21:16:36.474945+0000","last_fullsized":"2024-09-15T21:16:36.474945+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12525,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12808,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475374+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:36.475374+0000","last_peered":"2024-09-15T21:16:36.475374+0000","last_clean":"2024-09-15T21:16:36.475374+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:36.475374+0000","last_undegraded":"2024-09-15T21:16:36.475374+0000","last_fullsized":"2024-09-15T21:16:36.475374+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13791,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-15T21:16:36.475502+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:36.475502+0000","last_peered":"2024-09-15T21:16:36.475502+0000","last_clean":"2024-09-15T21:16:36.475502+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:36.475502+0000","last_undegraded":"2024-09-15T21:16:36.475502+0000","last_fullsized":"2024-09-15T21:16:36.475502+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":67900,"num_objects":10,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":97691,"num_write_kb":351354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4607,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":45056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149861,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708220,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663620,"statfs":{"total":96636764160,"available":95911546880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695224,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676616,"statfs":{"total":96636764160,"available":95924854784,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051737,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595728,"kb_used_data":448,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776112,"statfs":{"total":96636764160,"available":96026738688,"internally_reserved":0,"allocated":458752,"data_stored":218082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247975,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713720,"kb_used_data":420,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658120,"statfs":{"total":96636764160,"available":95905914880,"internally_reserved":0,"allocated":430080,"data_stored":189410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:52.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:52.822 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:52.822 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:52.835 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-15T21:16:52.229090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704628,"kb_used_data":1760,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782732,"statfs":{"total":386547056640,"available":383777517568,"internally_reserved":0,"allocated":1802240,"data_stored":808844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-577,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001335"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":13839,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:51.623266+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:51.623266+0000","last_peered":"2024-09-15T21:16:51.623266+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:51.623266+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587424,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784416,"statfs":{"total":96636764160,"available":96035241984,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713728,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658112,"statfs":{"total":96636764160,"available":95905906688,"internally_reserved":0,"allocated":438272,"data_stored":193223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:52.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:53.143 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:53.144 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:53.156 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-15T21:16:52.229090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704628,"kb_used_data":1760,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782732,"statfs":{"total":386547056640,"available":383777517568,"internally_reserved":0,"allocated":1802240,"data_stored":808844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-577,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001335"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":13839,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:51.623266+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:51.623266+0000","last_peered":"2024-09-15T21:16:51.623266+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:51.623266+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587424,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784416,"statfs":{"total":96636764160,"available":96035241984,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713728,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658112,"statfs":{"total":96636764160,"available":95905906688,"internally_reserved":0,"allocated":438272,"data_stored":193223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12556, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.511096+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:48.511096+0000', 'last_peered': '2024-09-15T21:16:48.511096+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:48.511096+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12757, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.512938+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:48.512938+0000', 'last_peered': '2024-09-15T21:16:48.512938+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:48.512938+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:53.157 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12465, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513553+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:48.513553+0000', 'last_peered': '2024-09-15T21:16:48.513553+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:48.513553+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:53.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:53.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12552, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513146+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:48.513146+0000', 'last_peered': '2024-09-15T21:16:48.513146+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:48.513146+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:53.158 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:53.158 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 13839, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:51.623266+0000', 'last_change': '2024-09-15T21:16:47.162979+0000', 'last_active': '2024-09-15T21:16:51.623266+0000', 'last_peered': '2024-09-15T21:16:51.623266+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:51.623266+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 665, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:53.158 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:53.541 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:53.541 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10816/32 objects degraded (33800.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":879,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":54320,"bytes_used":2769539072,"bytes_avail":383777517568,"bytes_total":386547056640,"degraded_objects":10816,"degraded_total":32,"degraded_ratio":338},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:53.555 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:53.555 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:53.862 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:53.862 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:53.875 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":483,"stamp":"2024-09-15T21:16:52.229090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704628,"kb_used_data":1760,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782732,"statfs":{"total":386547056640,"available":383777517568,"internally_reserved":0,"allocated":1802240,"data_stored":808844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-577,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001335"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":13839,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:51.623266+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:51.623266+0000","last_peered":"2024-09-15T21:16:51.623266+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:51.623266+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051738,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587424,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784416,"statfs":{"total":96636764160,"available":96035241984,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247976,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713728,"kb_used_data":428,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658112,"statfs":{"total":96636764160,"available":95905906688,"internally_reserved":0,"allocated":438272,"data_stored":193223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:56.877 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:57.181 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:57.181 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:57.194 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-15T21:16:56.229789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712952,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774408,"statfs":{"total":386547056640,"available":383768993792,"internally_reserved":0,"allocated":1810432,"data_stored":812636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.726682"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":13839,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:51.623266+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:51.623266+0000","last_peered":"2024-09-15T21:16:51.623266+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:51.623266+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595740,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776100,"statfs":{"total":96636764160,"available":96026726400,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:57.195 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:57.554 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:57.554 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:57.567 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2024-09-15T21:16:56.229789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712952,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774408,"statfs":{"total":386547056640,"available":383768993792,"internally_reserved":0,"allocated":1810432,"data_stored":812636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-679,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.726682"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":13839,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:51.623266+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:51.623266+0000","last_peered":"2024-09-15T21:16:51.623266+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:51.623266+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":665,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-1592}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2947,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595740,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776100,"statfs":{"total":96636764160,"available":96026726400,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12556, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.511096+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:48.511096+0000', 'last_peered': '2024-09-15T21:16:48.511096+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:48.511096+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12757, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.512938+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:48.512938+0000', 'last_peered': '2024-09-15T21:16:48.512938+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:48.512938+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:16:57.568 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12465, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513553+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:48.513553+0000', 'last_peered': '2024-09-15T21:16:48.513553+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:48.513553+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:57.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:16:57.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12552, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513146+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:48.513146+0000', 'last_peered': '2024-09-15T21:16:48.513146+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:48.513146+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:57.569 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:16:57.569 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 13839, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:51.623266+0000', 'last_change': '2024-09-15T21:16:47.162979+0000', 'last_active': '2024-09-15T21:16:51.623266+0000', 'last_peered': '2024-09-15T21:16:51.623266+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:51.623266+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1593, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 665, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -1592}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:16:57.569 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:16:57.964 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:57.964 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10816/32 objects degraded (33800.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":883,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":53,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":54320,"bytes_used":2778062848,"bytes_avail":383768993792,"bytes_total":386547056640,"degraded_objects":10816,"degraded_total":32,"degraded_ratio":338},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:16:57.978 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:16:57.978 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:16:58.290 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:16:58.290 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:16:58.304 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-15T21:16:58.230115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712952,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774408,"statfs":{"total":386547056640,"available":383768993792,"internally_reserved":0,"allocated":1810432,"data_stored":812636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12558,"num_write_kb":-50238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001661"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":15807,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:56.624843+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:56.624843+0000","last_peered":"2024-09-15T21:16:56.624843+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:56.624843+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595740,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776100,"statfs":{"total":96636764160,"available":96026726400,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:01.306 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:01.612 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:01.613 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:01.625 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-15T21:17:00.347456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712952,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774408,"statfs":{"total":386547056640,"available":383768993792,"internally_reserved":0,"allocated":1810432,"data_stored":812636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.118365"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":15807,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:56.624843+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:56.624843+0000","last_peered":"2024-09-15T21:16:56.624843+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:56.624843+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595740,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776100,"statfs":{"total":96636764160,"available":96026726400,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:01.626 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:01.933 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:01.934 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:01.947 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-15T21:17:00.347456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2712952,"kb_used_data":1768,"kb_used_omap":14,"kb_used_meta":2711089,"kb_avail":374774408,"statfs":{"total":386547056640,"available":383768993792,"internally_reserved":0,"allocated":1810432,"data_stored":812636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2776155698},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-984,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.118365"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17004,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513283+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:16:48.513283+0000","last_peered":"2024-09-15T21:16:48.513283+0000","last_clean":"2024-09-15T21:16:48.513283+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:16:48.513283+0000","last_undegraded":"2024-09-15T21:16:48.513283+0000","last_fullsized":"2024-09-15T21:16:48.513283+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17481,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517278+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:16:48.517278+0000","last_peered":"2024-09-15T21:16:48.517278+0000","last_clean":"2024-09-15T21:16:48.517278+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:16:48.517278+0000","last_undegraded":"2024-09-15T21:16:48.517278+0000","last_fullsized":"2024-09-15T21:16:48.517278+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":12556,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.511096+0000","last_change":"2024-09-15T21:03:59.729869+0000","last_active":"2024-09-15T21:16:48.511096+0000","last_peered":"2024-09-15T21:16:48.511096+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:16:48.511096+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-2257}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12570,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516916+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:16:48.516916+0000","last_peered":"2024-09-15T21:16:48.516916+0000","last_clean":"2024-09-15T21:16:48.516916+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:16:48.516916+0000","last_undegraded":"2024-09-15T21:16:48.516916+0000","last_fullsized":"2024-09-15T21:16:48.516916+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12757,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.512938+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:16:48.512938+0000","last_peered":"2024-09-15T21:16:48.512938+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:16:48.512938+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12465,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513553+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:16:48.513553+0000","last_peered":"2024-09-15T21:16:48.513553+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:16:48.513553+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12866,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516905+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:16:48.516905+0000","last_peered":"2024-09-15T21:16:48.516905+0000","last_clean":"2024-09-15T21:16:48.516905+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:16:48.516905+0000","last_undegraded":"2024-09-15T21:16:48.516905+0000","last_fullsized":"2024-09-15T21:16:48.516905+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12552,"reported_epoch":53,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:48.513146+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:16:48.513146+0000","last_peered":"2024-09-15T21:16:48.513146+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:16:48.513146+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17377,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.977744+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:16:48.977744+0000","last_peered":"2024-09-15T21:16:48.977744+0000","last_clean":"2024-09-15T21:16:48.977744+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:16:48.977744+0000","last_undegraded":"2024-09-15T21:16:48.977744+0000","last_fullsized":"2024-09-15T21:16:48.977744+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17067,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.513387+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:16:48.513387+0000","last_peered":"2024-09-15T21:16:48.513387+0000","last_clean":"2024-09-15T21:16:48.513387+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:16:48.513387+0000","last_undegraded":"2024-09-15T21:16:48.513387+0000","last_fullsized":"2024-09-15T21:16:48.513387+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":15807,"reported_epoch":53,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:16:56.624843+0000","last_change":"2024-09-15T21:16:47.162979+0000","last_active":"2024-09-15T21:16:56.624843+0000","last_peered":"2024-09-15T21:16:56.624843+0000","last_clean":"2024-09-15T21:02:42.003610+0000","last_became_active":"2024-09-15T21:03:59.672579+0000","last_became_peered":"2024-09-15T21:03:59.672579+0000","last_unstale":"2024-09-15T21:16:56.624843+0000","last_undegraded":"2024-09-15T21:03:59.103610+0000","last_fullsized":"2024-09-15T21:03:59.043106+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":609,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[2147483647,3,2,0],"avail_no_missing":["3(1)","0(3)","2(2)"],"object_location_counts":[{"shards":"0(3),2(2),3(1)","objects":-608}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17227,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516846+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:16:48.516846+0000","last_peered":"2024-09-15T21:16:48.516846+0000","last_clean":"2024-09-15T21:16:48.516846+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:16:48.516846+0000","last_undegraded":"2024-09-15T21:16:48.516846+0000","last_fullsized":"2024-09-15T21:16:48.516846+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12529,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516866+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:16:48.516866+0000","last_peered":"2024-09-15T21:16:48.516866+0000","last_clean":"2024-09-15T21:16:48.516866+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:16:48.516866+0000","last_undegraded":"2024-09-15T21:16:48.516866+0000","last_fullsized":"2024-09-15T21:16:48.516866+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12812,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.516954+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:16:48.516954+0000","last_peered":"2024-09-15T21:16:48.516954+0000","last_clean":"2024-09-15T21:16:48.516954+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:16:48.516954+0000","last_undegraded":"2024-09-15T21:16:48.516954+0000","last_fullsized":"2024-09-15T21:16:48.516954+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13795,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.517040+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:16:48.517040+0000","last_peered":"2024-09-15T21:16:48.517040+0000","last_clean":"2024-09-15T21:16:48.517040+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:16:48.517040+0000","last_undegraded":"2024-09-15T21:16:48.517040+0000","last_fullsized":"2024-09-15T21:16:48.517040+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9832,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":40960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149863,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708236,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663604,"statfs":{"total":96636764160,"available":95911530496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695240,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676600,"statfs":{"total":96636764160,"available":95924838400,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051739,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":595740,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":595259,"kb_avail":93776100,"statfs":{"total":96636764160,"available":96026726400,"internally_reserved":0,"allocated":471040,"data_stored":221591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":609545935},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:01.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:17:01.948 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 12556, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.511096+0000', 'last_change': '2024-09-15T21:03:59.729869+0000', 'last_active': '2024-09-15T21:16:48.511096+0000', 'last_peered': '2024-09-15T21:16:48.511096+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:16:48.511096+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2259, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -2257}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:17:01.948 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12757, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.512938+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:16:48.512938+0000', 'last_peered': '2024-09-15T21:16:48.512938+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:16:48.512938+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12465, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513553+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:16:48.513553+0000', 'last_peered': '2024-09-15T21:16:48.513553+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:16:48.513553+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12552, 'reported_epoch': 53, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:48.513146+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:16:48.513146+0000', 'last_peered': '2024-09-15T21:16:48.513146+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:16:48.513146+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.7 is not active+clean 2024-09-15T21:17:01.949 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.7', 'version': "37'12426", 'reported_seq': 15807, 'reported_epoch': 53, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:16:56.624843+0000', 'last_change': '2024-09-15T21:16:47.162979+0000', 'last_active': '2024-09-15T21:16:56.624843+0000', 'last_peered': '2024-09-15T21:16:56.624843+0000', 'last_clean': '2024-09-15T21:02:42.003610+0000', 'last_became_active': '2024-09-15T21:03:59.672579+0000', 'last_became_peered': '2024-09-15T21:03:59.672579+0000', 'last_unstale': '2024-09-15T21:16:56.624843+0000', 'last_undegraded': '2024-09-15T21:03:59.103610+0000', 'last_fullsized': '2024-09-15T21:03:59.043106+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10026, 'log_dups_size': 0, 'ondisk_log_size': 10026, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 609, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12425, 'num_write_kb': 49703, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1649, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3, 2, 0], 'acting': [2147483647, 3, 2, 0], 'avail_no_missing': ['3(1)', '0(3)', '2(2)'], 'object_location_counts': [{'shards': '0(3),2(2),3(1)', 'objects': -608}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:01.949 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:02.329 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:02.329 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9832/32 objects degraded (30725.000%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":888,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":8,"data_bytes":54320,"bytes_used":2778062848,"bytes_avail":383768993792,"bytes_total":386547056640,"degraded_objects":9832,"degraded_total":32,"degraded_ratio":307.25},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:02.345 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:02.346 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:02.648 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:02.648 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:02.660 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-15T21:17:02.230746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704668,"kb_used_data":1804,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782692,"statfs":{"total":386547056640,"available":383777476608,"internally_reserved":0,"allocated":1847296,"data_stored":827378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001283"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":13385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:02.107946+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:02.107946+0000","last_peered":"2024-09-15T21:17:02.107946+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:02.107946+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17638,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-15T21:16:48.515287+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:16:48.515287+0000","last_peered":"2024-09-15T21:16:48.515287+0000","last_clean":"2024-09-15T21:16:48.515287+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:16:48.515287+0000","last_undegraded":"2024-09-15T21:16:48.515287+0000","last_fullsized":"2024-09-15T21:16:48.515287+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12492,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.413982+0000","last_change":"2024-09-15T21:17:00.413480+0000","last_active":"2024-09-15T21:17:00.413982+0000","last_peered":"2024-09-15T21:17:00.413982+0000","last_clean":"2024-09-15T21:17:00.413982+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.413982+0000","last_undegraded":"2024-09-15T21:17:00.413982+0000","last_fullsized":"2024-09-15T21:17:00.413982+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587428,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784412,"statfs":{"total":96636764160,"available":96035237888,"internally_reserved":0,"allocated":475136,"data_stored":221271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247978,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713736,"kb_used_data":436,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658104,"statfs":{"total":96636764160,"available":95905898496,"internally_reserved":0,"allocated":446464,"data_stored":197015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:05.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:05.993 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:05.994 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:06.007 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-15T21:17:04.231133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2704688,"kb_used_data":1820,"kb_used_omap":14,"kb_used_meta":2702769,"kb_avail":374782672,"statfs":{"total":386547056640,"available":383777456128,"internally_reserved":0,"allocated":1863680,"data_stored":834909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2767636018},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2005,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001343"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":13385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:02.107946+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:02.107946+0000","last_peered":"2024-09-15T21:17:02.107946+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:02.107946+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12492,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.413982+0000","last_change":"2024-09-15T21:17:00.413480+0000","last_active":"2024-09-15T21:17:00.413982+0000","last_peered":"2024-09-15T21:17:00.413982+0000","last_clean":"2024-09-15T21:17:00.413982+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.413982+0000","last_undegraded":"2024-09-15T21:17:00.413982+0000","last_fullsized":"2024-09-15T21:17:00.413982+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":54320,"num_objects":8,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":85133,"num_write_kb":301116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2695,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051740,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":587428,"kb_used_data":464,"kb_used_omap":4,"kb_used_meta":586939,"kb_avail":93784412,"statfs":{"total":96636764160,"available":96035237888,"internally_reserved":0,"allocated":475136,"data_stored":221271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":601026255},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:06.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:06.336 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:06.337 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:06.349 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-15T21:17:06.231486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726900,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760460,"statfs":{"total":386547056640,"available":383754711040,"internally_reserved":0,"allocated":1871872,"data_stored":838664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12425,"num_write_kb":-49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001370"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":13385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:02.107946+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:02.107946+0000","last_peered":"2024-09-15T21:17:02.107946+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:02.107946+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609640,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762200,"statfs":{"total":96636764160,"available":96012492800,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 13385, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:02.107946+0000', 'last_change': '2024-09-15T21:16:59.297295+0000', 'last_active': '2024-09-15T21:17:02.107946+0000', 'last_peered': '2024-09-15T21:17:02.107946+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:17:02.107946+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1847, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 413, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1846}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12761, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.348691+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:17:00.348691+0000', 'last_peered': '2024-09-15T21:17:00.348691+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:00.348691+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12469, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.349141+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:17:00.349141+0000', 'last_peered': '2024-09-15T21:17:00.349141+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:17:00.349141+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:06.350 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:06.351 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12556, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.349142+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:00.349142+0000', 'last_peered': '2024-09-15T21:17:00.349142+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:00.349142+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:06.351 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:06.726 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:06.726 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 8811/28 objects degraded (31467.857%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":892,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47530,"bytes_used":2792345600,"bytes_avail":383754711040,"bytes_total":386547056640,"degraded_objects":8811,"degraded_total":28,"degraded_ratio":314.67857142857144},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:06.740 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:06.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:07.037 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:07.038 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:07.050 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":492,"stamp":"2024-09-15T21:17:06.231486+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726900,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760460,"statfs":{"total":386547056640,"available":383754711040,"internally_reserved":0,"allocated":1871872,"data_stored":838664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1021,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12425,"num_write_kb":-49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1237,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001370"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":13385,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:02.107946+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:02.107946+0000","last_peered":"2024-09-15T21:17:02.107946+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:02.107946+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1847,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1846}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609640,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762200,"statfs":{"total":96636764160,"available":96012492800,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247979,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:10.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:10.360 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:10.361 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:10.373 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-15T21:17:10.232182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726900,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760460,"statfs":{"total":386547056640,"available":383754711040,"internally_reserved":0,"allocated":1871872,"data_stored":838664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12425,"num_write_kb":-49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001745"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":15063,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:07.109862+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:07.109862+0000","last_peered":"2024-09-15T21:17:07.109862+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:07.109862+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609640,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762200,"statfs":{"total":96636764160,"available":96012492800,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:10.375 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:10.677 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:10.678 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:10.690 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-15T21:17:10.232182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726900,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760460,"statfs":{"total":386547056640,"available":383754711040,"internally_reserved":0,"allocated":1871872,"data_stored":838664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12425,"num_write_kb":-49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001745"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":15063,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:07.109862+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:07.109862+0000","last_peered":"2024-09-15T21:17:07.109862+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:07.109862+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609640,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762200,"statfs":{"total":96636764160,"available":96012492800,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 15063, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:07.109862+0000', 'last_change': '2024-09-15T21:16:59.297295+0000', 'last_active': '2024-09-15T21:17:07.109862+0000', 'last_peered': '2024-09-15T21:17:07.109862+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:17:07.109862+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1008, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1252, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': -1007}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12761, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.348691+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:17:00.348691+0000', 'last_peered': '2024-09-15T21:17:00.348691+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:00.348691+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:10.691 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12469, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.349141+0000', 'last_change': '2024-09-15T21:03:59.684771+0000', 'last_active': '2024-09-15T21:17:00.349141+0000', 'last_peered': '2024-09-15T21:17:00.349141+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:17:00.349141+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2316, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2316}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:10.692 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:10.692 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12556, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.349142+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:00.349142+0000', 'last_peered': '2024-09-15T21:17:00.349142+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:00.349142+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:10.692 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:11.081 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:11.081 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 7972/28 objects degraded (28471.429%), 4 pgs degraded, 4 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":896,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":55,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47530,"bytes_used":2792345600,"bytes_avail":383754711040,"bytes_total":386547056640,"degraded_objects":7972,"degraded_total":28,"degraded_ratio":284.71428571428572},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:11.094 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:11.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:11.407 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:11.408 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:11.421 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-15T21:17:10.232182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726900,"kb_used_data":1828,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760460,"statfs":{"total":386547056640,"available":383754711040,"internally_reserved":0,"allocated":1871872,"data_stored":838664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12425,"num_write_kb":-49703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-398,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001745"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":15063,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:07.109862+0000","last_change":"2024-09-15T21:16:59.297295+0000","last_active":"2024-09-15T21:17:07.109862+0000","last_peered":"2024-09-15T21:17:07.109862+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:07.109862+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1008,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1252,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":-1007}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12469,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349141+0000","last_change":"2024-09-15T21:03:59.684771+0000","last_active":"2024-09-15T21:17:00.349141+0000","last_peered":"2024-09-15T21:17:00.349141+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:00.349141+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17642,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348528+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:00.348528+0000","last_peered":"2024-09-15T21:17:00.348528+0000","last_clean":"2024-09-15T21:17:00.348528+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:00.348528+0000","last_undegraded":"2024-09-15T21:17:00.348528+0000","last_fullsized":"2024-09-15T21:17:00.348528+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7972,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051741,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609640,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762200,"statfs":{"total":96636764160,"available":96012492800,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247980,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713756,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658084,"statfs":{"total":96636764160,"available":95905878016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:14.423 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:14.774 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:14.775 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:14.787 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-15T21:17:14.232921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735224,"kb_used_data":1836,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752136,"statfs":{"total":386547056640,"available":383746187264,"internally_reserved":0,"allocated":1880064,"data_stored":842403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17080,"reported_epoch":55,"state":"active+undersized+remapped","last_fresh":"2024-09-15T21:17:11.526706+0000","last_change":"2024-09-15T21:17:11.526964+0000","last_active":"2024-09-15T21:17:11.526706+0000","last_peered":"2024-09-15T21:17:11.526706+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:11.526706+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12496,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:11.623863+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:11.623863+0000","last_peered":"2024-09-15T21:17:11.623863+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:11.623863+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617960,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753880,"statfs":{"total":96636764160,"available":96003973120,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713760,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658080,"statfs":{"total":96636764160,"available":95905873920,"internally_reserved":0,"allocated":471040,"data_stored":208285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:14.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:15.086 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:15.087 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:15.100 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-15T21:17:14.232921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735224,"kb_used_data":1836,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752136,"statfs":{"total":386547056640,"available":383746187264,"internally_reserved":0,"allocated":1880064,"data_stored":842403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17080,"reported_epoch":55,"state":"active+undersized+remapped","last_fresh":"2024-09-15T21:17:11.526706+0000","last_change":"2024-09-15T21:17:11.526964+0000","last_active":"2024-09-15T21:17:11.526706+0000","last_peered":"2024-09-15T21:17:11.526706+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:11.526706+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12496,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:11.623863+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:11.623863+0000","last_peered":"2024-09-15T21:17:11.623863+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:11.623863+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617960,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753880,"statfs":{"total":96636764160,"available":96003973120,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713760,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658080,"statfs":{"total":96636764160,"available":95905873920,"internally_reserved":0,"allocated":471040,"data_stored":208285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.d is not active+clean 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.d', 'version': "37'12432", 'reported_seq': 17080, 'reported_epoch': 55, 'state': 'active+undersized+remapped', 'last_fresh': '2024-09-15T21:17:11.526706+0000', 'last_change': '2024-09-15T21:17:11.526964+0000', 'last_active': '2024-09-15T21:17:11.526706+0000', 'last_peered': '2024-09-15T21:17:11.526706+0000', 'last_clean': '2024-09-15T21:02:50.468922+0000', 'last_became_active': '2024-09-15T21:03:59.715257+0000', 'last_became_peered': '2024-09-15T21:03:59.715257+0000', 'last_unstale': '2024-09-15T21:17:11.526706+0000', 'last_undegraded': '2024-09-15T21:03:59.088261+0000', 'last_fullsized': '2024-09-15T21:03:59.040855+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10032, 'log_dups_size': 0, 'ondisk_log_size': 10032, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 6790, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 1, 'num_write': 12431, 'num_write_kb': 49719, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2260, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1, 3, 0], 'acting': [2, 2147483647, 3, 0], 'avail_no_missing': ['2(0)', '0(3)', '1(1)', '3(2)'], 'object_location_counts': [{'shards': '0(3),2(0),3(2)', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12761, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.348691+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:17:00.348691+0000', 'last_peered': '2024-09-15T21:17:00.348691+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:00.348691+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:15.101 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 12496, 'reported_epoch': 55, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:11.623863+0000', 'last_change': '2024-09-15T21:17:11.527726+0000', 'last_active': '2024-09-15T21:17:11.623863+0000', 'last_peered': '2024-09-15T21:17:11.623863+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:17:11.623863+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2303, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 13, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2303}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:15.102 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:15.102 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12556, 'reported_epoch': 55, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:00.349142+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:00.349142+0000', 'last_peered': '2024-09-15T21:17:00.349142+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:00.349142+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:15.102 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:15.482 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:15.482 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6951/28 objects degraded (24825.000%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":901,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":12},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":7,"data_bytes":47530,"bytes_used":2800869376,"bytes_avail":383746187264,"bytes_total":386547056640,"degraded_objects":6951,"degraded_total":28,"degraded_ratio":248.25,"recovering_objects_per_sec":232,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1860,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:15.495 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:15.496 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:15.799 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:15.800 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:15.814 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":498,"stamp":"2024-09-15T21:17:14.232921+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735224,"kb_used_data":1836,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752136,"statfs":{"total":386547056640,"available":383746187264,"internally_reserved":0,"allocated":1880064,"data_stored":842403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1860,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001434"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17008,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.348745+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:00.348745+0000","last_peered":"2024-09-15T21:17:00.348745+0000","last_clean":"2024-09-15T21:17:00.348745+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:00.348745+0000","last_undegraded":"2024-09-15T21:17:00.348745+0000","last_fullsized":"2024-09-15T21:17:00.348745+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17485,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351424+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:00.351424+0000","last_peered":"2024-09-15T21:17:00.351424+0000","last_clean":"2024-09-15T21:17:00.351424+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:00.351424+0000","last_undegraded":"2024-09-15T21:17:00.351424+0000","last_fullsized":"2024-09-15T21:17:00.351424+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17080,"reported_epoch":55,"state":"active+undersized+remapped","last_fresh":"2024-09-15T21:17:11.526706+0000","last_change":"2024-09-15T21:17:11.526964+0000","last_active":"2024-09-15T21:17:11.526706+0000","last_peered":"2024-09-15T21:17:11.526706+0000","last_clean":"2024-09-15T21:02:50.468922+0000","last_became_active":"2024-09-15T21:03:59.715257+0000","last_became_peered":"2024-09-15T21:03:59.715257+0000","last_unstale":"2024-09-15T21:17:11.526706+0000","last_undegraded":"2024-09-15T21:03:59.088261+0000","last_fullsized":"2024-09-15T21:03:59.040855+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12431,"num_write_kb":49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,2147483647,3,0],"avail_no_missing":["2(0)","0(3)","1(1)","3(2)"],"object_location_counts":[{"shards":"0(3),2(0),3(2)","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12574,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351060+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:00.351060+0000","last_peered":"2024-09-15T21:17:00.351060+0000","last_clean":"2024-09-15T21:17:00.351060+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:00.351060+0000","last_undegraded":"2024-09-15T21:17:00.351060+0000","last_fullsized":"2024-09-15T21:17:00.351060+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12761,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.348691+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:00.348691+0000","last_peered":"2024-09-15T21:17:00.348691+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:00.348691+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":12496,"reported_epoch":55,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:11.623863+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:11.623863+0000","last_peered":"2024-09-15T21:17:11.623863+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:11.623863+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2303}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12870,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351284+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:00.351284+0000","last_peered":"2024-09-15T21:17:00.351284+0000","last_clean":"2024-09-15T21:17:00.351284+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:00.351284+0000","last_undegraded":"2024-09-15T21:17:00.351284+0000","last_fullsized":"2024-09-15T21:17:00.351284+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12556,"reported_epoch":55,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:00.349142+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:00.349142+0000","last_peered":"2024-09-15T21:17:00.349142+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:00.349142+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17381,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.350415+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:00.350415+0000","last_peered":"2024-09-15T21:17:00.350415+0000","last_clean":"2024-09-15T21:17:00.350415+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:00.350415+0000","last_undegraded":"2024-09-15T21:17:00.350415+0000","last_fullsized":"2024-09-15T21:17:00.350415+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17071,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.349412+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:00.349412+0000","last_peered":"2024-09-15T21:17:00.349412+0000","last_clean":"2024-09-15T21:17:00.349412+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:00.349412+0000","last_undegraded":"2024-09-15T21:17:00.349412+0000","last_fullsized":"2024-09-15T21:17:00.349412+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12500,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.883953+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:00.883953+0000","last_peered":"2024-09-15T21:17:00.883953+0000","last_clean":"2024-09-15T21:17:00.883953+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:00.883953+0000","last_undegraded":"2024-09-15T21:17:00.883953+0000","last_fullsized":"2024-09-15T21:17:00.883953+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17231,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351113+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:00.351113+0000","last_peered":"2024-09-15T21:17:00.351113+0000","last_clean":"2024-09-15T21:17:00.351113+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:00.351113+0000","last_undegraded":"2024-09-15T21:17:00.351113+0000","last_fullsized":"2024-09-15T21:17:00.351113+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12533,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351192+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:00.351192+0000","last_peered":"2024-09-15T21:17:00.351192+0000","last_clean":"2024-09-15T21:17:00.351192+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:00.351192+0000","last_undegraded":"2024-09-15T21:17:00.351192+0000","last_fullsized":"2024-09-15T21:17:00.351192+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12816,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351405+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:00.351405+0000","last_peered":"2024-09-15T21:17:00.351405+0000","last_clean":"2024-09-15T21:17:00.351405+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:00.351405+0000","last_undegraded":"2024-09-15T21:17:00.351405+0000","last_fullsized":"2024-09-15T21:17:00.351405+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13799,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-09-15T21:17:00.351310+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:00.351310+0000","last_peered":"2024-09-15T21:17:00.351310+0000","last_clean":"2024-09-15T21:17:00.351310+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:00.351310+0000","last_undegraded":"2024-09-15T21:17:00.351310+0000","last_fullsized":"2024-09-15T21:17:00.351310+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":47530,"num_objects":7,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":72708,"num_write_kb":251413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149866,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708248,"kb_used_data":452,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663592,"statfs":{"total":96636764160,"available":95911518208,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695256,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676584,"statfs":{"total":96636764160,"available":95924822016,"internally_reserved":0,"allocated":462848,"data_stored":204546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051742,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617960,"kb_used_data":472,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753880,"statfs":{"total":96636764160,"available":96003973120,"internally_reserved":0,"allocated":483328,"data_stored":225026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713760,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658080,"statfs":{"total":96636764160,"available":95905873920,"internally_reserved":0,"allocated":471040,"data_stored":208285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:18.816 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:19.190 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:19.191 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:19.205 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-15T21:17:18.233581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726968,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760392,"statfs":{"total":386547056640,"available":383754641408,"internally_reserved":0,"allocated":1929216,"data_stored":864774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001399"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":14114,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:16.621938+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:16.621938+0000","last_peered":"2024-09-15T21:17:16.621938+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:16.621938+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":820,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17385,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388445+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388445+0000","last_peered":"2024-09-15T21:17:13.388445+0000","last_clean":"2024-09-15T21:17:13.388445+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388445+0000","last_undegraded":"2024-09-15T21:17:13.388445+0000","last_fullsized":"2024-09-15T21:17:13.388445+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713760,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658080,"statfs":{"total":96636764160,"available":95905873920,"internally_reserved":0,"allocated":471040,"data_stored":208285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:19.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:19.519 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:19.520 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:19.532 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":500,"stamp":"2024-09-15T21:17:18.233581+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726968,"kb_used_data":1884,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760392,"statfs":{"total":386547056640,"available":383754641408,"internally_reserved":0,"allocated":1929216,"data_stored":864774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-432,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001399"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":14114,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:16.621938+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:16.621938+0000","last_peered":"2024-09-15T21:17:16.621938+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:16.621938+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":820,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17385,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388445+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388445+0000","last_peered":"2024-09-15T21:17:13.388445+0000","last_clean":"2024-09-15T21:17:13.388445+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388445+0000","last_undegraded":"2024-09-15T21:17:13.388445+0000","last_fullsized":"2024-09-15T21:17:13.388445+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247981,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713760,"kb_used_data":460,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658080,"statfs":{"total":96636764160,"available":95905873920,"internally_reserved":0,"allocated":471040,"data_stored":208285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:19.534 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:19.534 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12765, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:13.389563+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:17:13.389563+0000', 'last_peered': '2024-09-15T21:17:13.389563+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:13.389563+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:19.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:19.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 14114, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:16.621938+0000', 'last_change': '2024-09-15T21:17:11.527726+0000', 'last_active': '2024-09-15T21:17:16.621938+0000', 'last_peered': '2024-09-15T21:17:16.621938+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:17:16.621938+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1496, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 820, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1496}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:19.535 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:19.535 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12560, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:13.389594+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:13.389594+0000', 'last_peered': '2024-09-15T21:17:13.389594+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:13.389594+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:19.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:19.957 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:19.958 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6144/24 objects degraded (25600.000%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":905,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2792415232,"bytes_avail":383754641408,"bytes_total":386547056640,"degraded_objects":6144,"degraded_total":24,"degraded_ratio":256},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:19.970 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:19.971 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:20.271 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:20.271 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:20.284 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-15T21:17:20.233889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726976,"kb_used_data":1892,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760384,"statfs":{"total":386547056640,"available":383754633216,"internally_reserved":0,"allocated":1937408,"data_stored":868492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-807,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1453,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001318"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":14114,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:16.621938+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:16.621938+0000","last_peered":"2024-09-15T21:17:16.621938+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:16.621938+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1496,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":820,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1496}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17385,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388445+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388445+0000","last_peered":"2024-09-15T21:17:13.388445+0000","last_clean":"2024-09-15T21:17:13.388445+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388445+0000","last_undegraded":"2024-09-15T21:17:13.388445+0000","last_fullsized":"2024-09-15T21:17:13.388445+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6144,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149867,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051743,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:23.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:23.611 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:23.611 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:23.624 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-15T21:17:22.234179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735288,"kb_used_data":1892,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752072,"statfs":{"total":386547056640,"available":383746121728,"internally_reserved":0,"allocated":1937408,"data_stored":868492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.856597"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":16024,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:21.622411+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:21.622411+0000","last_peered":"2024-09-15T21:17:21.622411+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:21.622411+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17386,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388493+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388493+0000","last_peered":"2024-09-15T21:17:13.388493+0000","last_clean":"2024-09-15T21:17:13.388493+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388493+0000","last_undegraded":"2024-09-15T21:17:13.388493+0000","last_fullsized":"2024-09-15T21:17:13.388493+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617976,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753864,"statfs":{"total":96636764160,"available":96003956736,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:23.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:23.925 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:23.926 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:23.940 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-15T21:17:22.234179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735288,"kb_used_data":1892,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752072,"statfs":{"total":386547056640,"available":383746121728,"internally_reserved":0,"allocated":1937408,"data_stored":868492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.856597"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":16024,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:21.622411+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:21.622411+0000","last_peered":"2024-09-15T21:17:21.622411+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:21.622411+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17386,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388493+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388493+0000","last_peered":"2024-09-15T21:17:13.388493+0000","last_clean":"2024-09-15T21:17:13.388493+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388493+0000","last_undegraded":"2024-09-15T21:17:13.388493+0000","last_fullsized":"2024-09-15T21:17:13.388493+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617976,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753864,"statfs":{"total":96636764160,"available":96003956736,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247982,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:23.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:23.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 12765, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:13.389563+0000', 'last_change': '2024-09-15T21:03:59.725695+0000', 'last_active': '2024-09-15T21:17:13.389563+0000', 'last_peered': '2024-09-15T21:17:13.389563+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:13.389563+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2313, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -2309}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:23.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.c is not active+clean 2024-09-15T21:17:23.941 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.c', 'version': "37'12376", 'reported_seq': 16024, 'reported_epoch': 57, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:21.622411+0000', 'last_change': '2024-09-15T21:17:11.527726+0000', 'last_active': '2024-09-15T21:17:21.622411+0000', 'last_peered': '2024-09-15T21:17:21.622411+0000', 'last_clean': '2024-09-15T21:02:38.968547+0000', 'last_became_active': '2024-09-15T21:03:59.675247+0000', 'last_became_peered': '2024-09-15T21:03:59.675247+0000', 'last_unstale': '2024-09-15T21:17:21.622411+0000', 'last_undegraded': '2024-09-15T21:03:59.119872+0000', 'last_fullsized': '2024-09-15T21:03:59.042715+0000', 'mapping_epoch': 30, 'log_start': "28'2300", 'ondisk_log_start': "28'2300", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10076, 'log_dups_size': 0, 'ondisk_log_size': 10076, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 541, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12376, 'num_write_kb': 49504, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1775, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -541}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:23.941 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:23.942 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12560, 'reported_epoch': 57, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:13.389594+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:13.389594+0000', 'last_peered': '2024-09-15T21:17:13.389594+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:13.389594+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:23.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:24.328 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5189/24 objects degraded (21620.833%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":910,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":57,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":13},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2800934912,"bytes_avail":383746121728,"bytes_total":386547056640,"degraded_objects":5189,"degraded_total":24,"degraded_ratio":216.20833333333337},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:24.341 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:24.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:24.703 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:24.704 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:24.717 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-15T21:17:24.350049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735288,"kb_used_data":1892,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752072,"statfs":{"total":386547056640,"available":383746121728,"internally_reserved":0,"allocated":1937408,"data_stored":868492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6790,"num_objects":-1,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-12431,"num_write_kb":-49719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-498,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.117128"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17489,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390747+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:13.390747+0000","last_peered":"2024-09-15T21:17:13.390747+0000","last_clean":"2024-09-15T21:17:13.390747+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:13.390747+0000","last_undegraded":"2024-09-15T21:17:13.390747+0000","last_fullsized":"2024-09-15T21:17:13.390747+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17100,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:14.072679+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:14.072679+0000","last_peered":"2024-09-15T21:17:14.072679+0000","last_clean":"2024-09-15T21:17:14.072679+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:14.072679+0000","last_undegraded":"2024-09-15T21:17:14.072679+0000","last_fullsized":"2024-09-15T21:17:14.072679+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12578,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390300+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:13.390300+0000","last_peered":"2024-09-15T21:17:13.390300+0000","last_clean":"2024-09-15T21:17:13.390300+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:13.390300+0000","last_undegraded":"2024-09-15T21:17:13.390300+0000","last_fullsized":"2024-09-15T21:17:13.390300+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":16024,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:21.622411+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:21.622411+0000","last_peered":"2024-09-15T21:17:21.622411+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:21.622411+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12874,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390496+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:13.390496+0000","last_peered":"2024-09-15T21:17:13.390496+0000","last_clean":"2024-09-15T21:17:13.390496+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:13.390496+0000","last_undegraded":"2024-09-15T21:17:13.390496+0000","last_fullsized":"2024-09-15T21:17:13.390496+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17386,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.388493+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:13.388493+0000","last_peered":"2024-09-15T21:17:13.388493+0000","last_clean":"2024-09-15T21:17:13.388493+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:13.388493+0000","last_undegraded":"2024-09-15T21:17:13.388493+0000","last_fullsized":"2024-09-15T21:17:13.388493+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12504,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390822+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:13.390822+0000","last_peered":"2024-09-15T21:17:13.390822+0000","last_clean":"2024-09-15T21:17:13.390822+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:13.390822+0000","last_undegraded":"2024-09-15T21:17:13.390822+0000","last_fullsized":"2024-09-15T21:17:13.390822+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17235,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390387+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:13.390387+0000","last_peered":"2024-09-15T21:17:13.390387+0000","last_clean":"2024-09-15T21:17:13.390387+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:13.390387+0000","last_undegraded":"2024-09-15T21:17:13.390387+0000","last_fullsized":"2024-09-15T21:17:13.390387+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12537,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390281+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:13.390281+0000","last_peered":"2024-09-15T21:17:13.390281+0000","last_clean":"2024-09-15T21:17:13.390281+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:13.390281+0000","last_undegraded":"2024-09-15T21:17:13.390281+0000","last_fullsized":"2024-09-15T21:17:13.390281+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12820,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390430+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:13.390430+0000","last_peered":"2024-09-15T21:17:13.390430+0000","last_clean":"2024-09-15T21:17:13.390430+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:13.390430+0000","last_undegraded":"2024-09-15T21:17:13.390430+0000","last_fullsized":"2024-09-15T21:17:13.390430+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13803,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.390589+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:13.390589+0000","last_peered":"2024-09-15T21:17:13.390589+0000","last_clean":"2024-09-15T21:17:13.390589+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:13.390589+0000","last_undegraded":"2024-09-15T21:17:13.390589+0000","last_fullsized":"2024-09-15T21:17:13.390589+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":36864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695276,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676564,"statfs":{"total":96636764160,"available":95924801536,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051744,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617976,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753864,"statfs":{"total":96636764160,"available":96003956736,"internally_reserved":0,"allocated":499712,"data_stored":232483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:27.720 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:28.069 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:28.069 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:28.083 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-15T21:17:26.234852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726976,"kb_used_data":1900,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760384,"statfs":{"total":386547056640,"available":383754633216,"internally_reserved":0,"allocated":1945600,"data_stored":867704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":955,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001270"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17012,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389845+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:13.389845+0000","last_peered":"2024-09-15T21:17:13.389845+0000","last_clean":"2024-09-15T21:17:13.389845+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:13.389845+0000","last_undegraded":"2024-09-15T21:17:13.389845+0000","last_fullsized":"2024-09-15T21:17:13.389845+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":12765,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389563+0000","last_change":"2024-09-15T21:03:59.725695+0000","last_active":"2024-09-15T21:17:13.389563+0000","last_peered":"2024-09-15T21:17:13.389563+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:13.389563+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-2309}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":16024,"reported_epoch":57,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:21.622411+0000","last_change":"2024-09-15T21:17:11.527726+0000","last_active":"2024-09-15T21:17:21.622411+0000","last_peered":"2024-09-15T21:17:21.622411+0000","last_clean":"2024-09-15T21:02:38.968547+0000","last_became_active":"2024-09-15T21:03:59.675247+0000","last_became_peered":"2024-09-15T21:03:59.675247+0000","last_unstale":"2024-09-15T21:17:21.622411+0000","last_undegraded":"2024-09-15T21:03:59.119872+0000","last_fullsized":"2024-09-15T21:03:59.042715+0000","mapping_epoch":30,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":541,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1775,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-541}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12560,"reported_epoch":57,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:13.389594+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:13.389594+0000","last_peered":"2024-09-15T21:17:13.389594+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:13.389594+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17075,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.389700+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:13.389700+0000","last_peered":"2024-09-15T21:17:13.389700+0000","last_clean":"2024-09-15T21:17:13.389700+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:13.389700+0000","last_undegraded":"2024-09-15T21:17:13.389700+0000","last_fullsized":"2024-09-15T21:17:13.389700+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4056,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149868,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708268,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663572,"statfs":{"total":96636764160,"available":95911497728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695280,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676560,"statfs":{"total":96636764160,"available":95924797440,"internally_reserved":0,"allocated":487424,"data_stored":215705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":499712,"data_stored":227993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:28.084 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:28.419 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:28.419 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:28.432 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-15T21:17:28.235185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726992,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760368,"statfs":{"total":386547056640,"available":383754616832,"internally_reserved":0,"allocated":1961984,"data_stored":875087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001295"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":13740,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:26.626498+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:26.626498+0000","last_peered":"2024-09-15T21:17:26.626498+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:26.626498+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695280,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676560,"statfs":{"total":96636764160,"available":95924797440,"internally_reserved":0,"allocated":487424,"data_stored":215705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":499712,"data_stored":227993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:28.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:28.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 13740, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:26.626498+0000', 'last_change': '2024-09-15T21:17:23.450047+0000', 'last_active': '2024-09-15T21:17:26.626498+0000', 'last_peered': '2024-09-15T21:17:26.626498+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:26.626498+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1828, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 487, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -1826}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:28.433 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:28.433 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12564, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:25.353684+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:25.353684+0000', 'last_peered': '2024-09-15T21:17:25.353684+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:25.353684+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:28.433 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:28.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:28.832 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4163/24 objects degraded (17345.833%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":914,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2792439808,"bytes_avail":383754616832,"bytes_total":386547056640,"degraded_objects":4163,"degraded_total":24,"degraded_ratio":173.45833333333337,"recovering_objects_per_sec":247,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1981,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:28.846 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:28.847 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:29.223 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:29.223 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:29.236 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-15T21:17:28.235185+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2726992,"kb_used_data":1916,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760368,"statfs":{"total":386547056640,"available":383754616832,"internally_reserved":0,"allocated":1961984,"data_stored":875087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1981,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1981,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001295"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":13740,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:26.626498+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:26.626498+0000","last_peered":"2024-09-15T21:17:26.626498+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:26.626498+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1828,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":487,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-1826}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17646,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-15T21:17:13.387755+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:13.387755+0000","last_peered":"2024-09-15T21:17:13.387755+0000","last_clean":"2024-09-15T21:17:13.387755+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:13.387755+0000","last_undegraded":"2024-09-15T21:17:13.387755+0000","last_fullsized":"2024-09-15T21:17:13.387755+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4163,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5082,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695280,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676560,"statfs":{"total":96636764160,"available":95924797440,"internally_reserved":0,"allocated":487424,"data_stored":215705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051745,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609660,"kb_used_data":488,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762180,"statfs":{"total":96636764160,"available":96012472320,"internally_reserved":0,"allocated":499712,"data_stored":227993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247983,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713768,"kb_used_data":468,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658072,"statfs":{"total":96636764160,"available":95905865728,"internally_reserved":0,"allocated":479232,"data_stored":212003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:32.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:32.564 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:32.565 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:32.578 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T21:17:32.235854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735344,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752016,"statfs":{"total":386547056640,"available":383746064384,"internally_reserved":0,"allocated":1994752,"data_stored":889832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001318"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":15419,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:31.627958+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:31.627958+0000","last_peered":"2024-09-15T21:17:31.627958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:31.627958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617984,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753856,"statfs":{"total":96636764160,"available":96003948544,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:32.580 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:32.880 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:32.880 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:32.893 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T21:17:32.235854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735344,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752016,"statfs":{"total":386547056640,"available":383746064384,"internally_reserved":0,"allocated":1994752,"data_stored":889832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001318"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":15419,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:31.627958+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:31.627958+0000","last_peered":"2024-09-15T21:17:31.627958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:31.627958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617984,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753856,"statfs":{"total":96636764160,"available":96003948544,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:32.894 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:32.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 15419, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:31.627958+0000', 'last_change': '2024-09-15T21:17:23.450047+0000', 'last_active': '2024-09-15T21:17:31.627958+0000', 'last_peered': '2024-09-15T21:17:31.627958+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:31.627958+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -987}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:32.895 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:32.895 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12564, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:25.353684+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:25.353684+0000', 'last_peered': '2024-09-15T21:17:25.353684+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:25.353684+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:32.895 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:33.307 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:33.307 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3324/24 objects degraded (13850.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":919,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2800992256,"bytes_avail":383746064384,"bytes_total":386547056640,"degraded_objects":3324,"degraded_total":24,"degraded_ratio":138.5,"recovering_objects_per_sec":233,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:33.322 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:33.323 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:33.654 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:33.654 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:33.667 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2024-09-15T21:17:32.235854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735344,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374752016,"statfs":{"total":386547056640,"available":383746064384,"internally_reserved":0,"allocated":1994752,"data_stored":889832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001318"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17493,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353536+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353536+0000","last_peered":"2024-09-15T21:17:25.353536+0000","last_clean":"2024-09-15T21:17:25.353536+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353536+0000","last_undegraded":"2024-09-15T21:17:25.353536+0000","last_fullsized":"2024-09-15T21:17:25.353536+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":15419,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:31.627958+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:31.627958+0000","last_peered":"2024-09-15T21:17:31.627958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:31.627958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051746,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617984,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753856,"statfs":{"total":96636764160,"available":96003948544,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247984,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:36.671 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:37.007 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:37.008 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:37.021 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-15T21:17:36.451247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727024,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760336,"statfs":{"total":386547056640,"available":383754584064,"internally_reserved":0,"allocated":1994752,"data_stored":889832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.216395"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17494,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353581+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353581+0000","last_peered":"2024-09-15T21:17:25.353581+0000","last_clean":"2024-09-15T21:17:25.353581+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353581+0000","last_undegraded":"2024-09-15T21:17:25.353581+0000","last_fullsized":"2024-09-15T21:17:25.353581+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":15419,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:31.627958+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:31.627958+0000","last_peered":"2024-09-15T21:17:31.627958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:31.627958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:37.022 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:37.383 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:37.384 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:37.397 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-15T21:17:36.451247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727024,"kb_used_data":1948,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760336,"statfs":{"total":386547056640,"available":383754584064,"internally_reserved":0,"allocated":1994752,"data_stored":889832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1865,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.216395"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17016,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353957+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:25.353957+0000","last_peered":"2024-09-15T21:17:25.353957+0000","last_clean":"2024-09-15T21:17:25.353957+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:25.353957+0000","last_undegraded":"2024-09-15T21:17:25.353957+0000","last_fullsized":"2024-09-15T21:17:25.353957+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17494,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353581+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353581+0000","last_peered":"2024-09-15T21:17:25.353581+0000","last_clean":"2024-09-15T21:17:25.353581+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353581+0000","last_undegraded":"2024-09-15T21:17:25.353581+0000","last_fullsized":"2024-09-15T21:17:25.353581+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":15419,"reported_epoch":59,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:31.627958+0000","last_change":"2024-09-15T21:17:23.450047+0000","last_active":"2024-09-15T21:17:31.627958+0000","last_peered":"2024-09-15T21:17:31.627958+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:31.627958+0000","last_undegraded":"2024-09-15T21:03:59.115088+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":30,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":989,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1326,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,2147483647,0,2],"avail_no_missing":["3(0)","0(2)","2(3)"],"object_location_counts":[{"shards":"0(2),2(3),3(0)","objects":-987}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17119,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.437683+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:25.437683+0000","last_peered":"2024-09-15T21:17:25.437683+0000","last_clean":"2024-09-15T21:17:25.437683+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:25.437683+0000","last_undegraded":"2024-09-15T21:17:25.437683+0000","last_fullsized":"2024-09-15T21:17:25.437683+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12564,"reported_epoch":59,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:25.353684+0000","last_change":"2024-09-15T21:03:59.731114+0000","last_active":"2024-09-15T21:17:25.353684+0000","last_peered":"2024-09-15T21:17:25.353684+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:25.353684+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2335,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2328}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17079,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.354366+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:25.354366+0000","last_peered":"2024-09-15T21:17:25.354366+0000","last_clean":"2024-09-15T21:17:25.354366+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:25.354366+0000","last_undegraded":"2024-09-15T21:17:25.354366+0000","last_fullsized":"2024-09-15T21:17:25.354366+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5921,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149870,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708284,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663556,"statfs":{"total":96636764160,"available":95911481344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:37.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:37.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 15419, 'reported_epoch': 59, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:31.627958+0000', 'last_change': '2024-09-15T21:17:23.450047+0000', 'last_active': '2024-09-15T21:17:31.627958+0000', 'last_peered': '2024-09-15T21:17:31.627958+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:31.627958+0000', 'last_undegraded': '2024-09-15T21:03:59.115088+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 30, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 989, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1326, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 2147483647, 0, 2], 'avail_no_missing': ['3(0)', '0(2)', '2(3)'], 'object_location_counts': [{'shards': '0(2),2(3),3(0)', 'objects': -987}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:37.398 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:37.398 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12564, 'reported_epoch': 59, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:25.353684+0000', 'last_change': '2024-09-15T21:03:59.731114+0000', 'last_active': '2024-09-15T21:17:25.353684+0000', 'last_peered': '2024-09-15T21:17:25.353684+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:25.353684+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2335, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2328}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:37.398 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:37.816 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:37.817 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3324/24 objects degraded (13850.000%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":923,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2792472576,"bytes_avail":383754584064,"bytes_total":386547056640,"degraded_objects":3324,"degraded_total":24,"degraded_ratio":138.5,"recovering_objects_per_sec":171,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1865,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:37.831 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:37.831 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:38.189 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:38.190 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:38.205 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-15T21:17:37.454790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727028,"kb_used_data":1956,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760332,"statfs":{"total":386547056640,"available":383754579968,"internally_reserved":0,"allocated":2002944,"data_stored":893497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1963,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.219605"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17018,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454013+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:36.454013+0000","last_peered":"2024-09-15T21:17:36.454013+0000","last_clean":"2024-09-15T21:17:36.454013+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:36.454013+0000","last_undegraded":"2024-09-15T21:17:36.454013+0000","last_fullsized":"2024-09-15T21:17:36.454013+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17494,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353581+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353581+0000","last_peered":"2024-09-15T21:17:25.353581+0000","last_clean":"2024-09-15T21:17:25.353581+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353581+0000","last_undegraded":"2024-09-15T21:17:25.353581+0000","last_fullsized":"2024-09-15T21:17:25.353581+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17399,"reported_epoch":59,"state":"peering","last_fresh":"2024-09-15T21:17:35.794409+0000","last_change":"2024-09-15T21:17:36.457811+0000","last_active":"2024-09-15T21:17:36.457811+0000","last_peered":"2024-09-15T21:17:35.794409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:35.794409+0000","last_undegraded":"2024-09-15T21:17:35.794409+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454102+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:36.454102+0000","last_peered":"2024-09-15T21:17:36.454102+0000","last_clean":"2024-09-15T21:17:36.454102+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:36.454102+0000","last_undegraded":"2024-09-15T21:17:36.454102+0000","last_fullsized":"2024-09-15T21:17:36.454102+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17650,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352935+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:25.352935+0000","last_peered":"2024-09-15T21:17:25.352935+0000","last_clean":"2024-09-15T21:17:25.352935+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:25.352935+0000","last_undegraded":"2024-09-15T21:17:25.352935+0000","last_fullsized":"2024-09-15T21:17:25.352935+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12837,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:36.627009+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:36.627009+0000","last_peered":"2024-09-15T21:17:36.627009+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:36.627009+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17081,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454103+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:36.454103+0000","last_peered":"2024-09-15T21:17:36.454103+0000","last_clean":"2024-09-15T21:17:36.454103+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:36.454103+0000","last_undegraded":"2024-09-15T21:17:36.454103+0000","last_fullsized":"2024-09-15T21:17:36.454103+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708288,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663552,"statfs":{"total":96636764160,"available":95911477248,"internally_reserved":0,"allocated":503808,"data_stored":223051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247985,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713784,"kb_used_data":484,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658056,"statfs":{"total":96636764160,"available":95905849344,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:41.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:41.519 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:41.519 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:41.533 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-15T21:17:40.237148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727044,"kb_used_data":1972,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760316,"statfs":{"total":386547056640,"available":383754563584,"internally_reserved":0,"allocated":2019328,"data_stored":900806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001293"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17018,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454013+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:36.454013+0000","last_peered":"2024-09-15T21:17:36.454013+0000","last_clean":"2024-09-15T21:17:36.454013+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:36.454013+0000","last_undegraded":"2024-09-15T21:17:36.454013+0000","last_fullsized":"2024-09-15T21:17:36.454013+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17494,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353581+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353581+0000","last_peered":"2024-09-15T21:17:25.353581+0000","last_clean":"2024-09-15T21:17:25.353581+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353581+0000","last_undegraded":"2024-09-15T21:17:25.353581+0000","last_fullsized":"2024-09-15T21:17:25.353581+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17399,"reported_epoch":59,"state":"peering","last_fresh":"2024-09-15T21:17:35.794409+0000","last_change":"2024-09-15T21:17:36.457811+0000","last_active":"2024-09-15T21:17:36.457811+0000","last_peered":"2024-09-15T21:17:35.794409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:35.794409+0000","last_undegraded":"2024-09-15T21:17:35.794409+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454102+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:36.454102+0000","last_peered":"2024-09-15T21:17:36.454102+0000","last_clean":"2024-09-15T21:17:36.454102+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:36.454102+0000","last_undegraded":"2024-09-15T21:17:36.454102+0000","last_fullsized":"2024-09-15T21:17:36.454102+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17654,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460351+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460351+0000","last_peered":"2024-09-15T21:17:37.460351+0000","last_clean":"2024-09-15T21:17:37.460351+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460351+0000","last_undegraded":"2024-09-15T21:17:37.460351+0000","last_fullsized":"2024-09-15T21:17:37.460351+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12837,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:36.627009+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:36.627009+0000","last_peered":"2024-09-15T21:17:36.627009+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:36.627009+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17081,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454103+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:36.454103+0000","last_peered":"2024-09-15T21:17:36.454103+0000","last_clean":"2024-09-15T21:17:36.454103+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:36.454103+0000","last_undegraded":"2024-09-15T21:17:36.454103+0000","last_fullsized":"2024-09-15T21:17:36.454103+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708288,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663552,"statfs":{"total":96636764160,"available":95911477248,"internally_reserved":0,"allocated":503808,"data_stored":223051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:41.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:41.841 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:41.842 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:41.854 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-15T21:17:40.237148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727044,"kb_used_data":1972,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760316,"statfs":{"total":386547056640,"available":383754563584,"internally_reserved":0,"allocated":2019328,"data_stored":900806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001293"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17018,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454013+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:36.454013+0000","last_peered":"2024-09-15T21:17:36.454013+0000","last_clean":"2024-09-15T21:17:36.454013+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:36.454013+0000","last_undegraded":"2024-09-15T21:17:36.454013+0000","last_fullsized":"2024-09-15T21:17:36.454013+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17494,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353581+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:25.353581+0000","last_peered":"2024-09-15T21:17:25.353581+0000","last_clean":"2024-09-15T21:17:25.353581+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:25.353581+0000","last_undegraded":"2024-09-15T21:17:25.353581+0000","last_fullsized":"2024-09-15T21:17:25.353581+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17104,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355812+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:25.355812+0000","last_peered":"2024-09-15T21:17:25.355812+0000","last_clean":"2024-09-15T21:17:25.355812+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:25.355812+0000","last_undegraded":"2024-09-15T21:17:25.355812+0000","last_fullsized":"2024-09-15T21:17:25.355812+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12582,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352953+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:25.352953+0000","last_peered":"2024-09-15T21:17:25.352953+0000","last_clean":"2024-09-15T21:17:25.352953+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:25.352953+0000","last_undegraded":"2024-09-15T21:17:25.352953+0000","last_fullsized":"2024-09-15T21:17:25.352953+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17399,"reported_epoch":59,"state":"peering","last_fresh":"2024-09-15T21:17:35.794409+0000","last_change":"2024-09-15T21:17:36.457811+0000","last_active":"2024-09-15T21:17:36.457811+0000","last_peered":"2024-09-15T21:17:35.794409+0000","last_clean":"2024-09-15T21:02:49.753923+0000","last_became_active":"2024-09-15T21:03:59.704275+0000","last_became_peered":"2024-09-15T21:03:59.704275+0000","last_unstale":"2024-09-15T21:17:35.794409+0000","last_undegraded":"2024-09-15T21:17:35.794409+0000","last_fullsized":"2024-09-15T21:03:59.043397+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13580,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12655,"num_write_kb":50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2315,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17121,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454102+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:36.454102+0000","last_peered":"2024-09-15T21:17:36.454102+0000","last_clean":"2024-09-15T21:17:36.454102+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:36.454102+0000","last_undegraded":"2024-09-15T21:17:36.454102+0000","last_fullsized":"2024-09-15T21:17:36.454102+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17654,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460351+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460351+0000","last_peered":"2024-09-15T21:17:37.460351+0000","last_clean":"2024-09-15T21:17:37.460351+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460351+0000","last_undegraded":"2024-09-15T21:17:37.460351+0000","last_fullsized":"2024-09-15T21:17:37.460351+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12878,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353817+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:25.353817+0000","last_peered":"2024-09-15T21:17:25.353817+0000","last_clean":"2024-09-15T21:17:25.353817+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:25.353817+0000","last_undegraded":"2024-09-15T21:17:25.353817+0000","last_fullsized":"2024-09-15T21:17:25.353817+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":12837,"reported_epoch":60,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:36.627009+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:36.627009+0000","last_peered":"2024-09-15T21:17:36.627009+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:36.627009+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":138,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-2196}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17390,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.355833+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:25.355833+0000","last_peered":"2024-09-15T21:17:25.355833+0000","last_clean":"2024-09-15T21:17:25.355833+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:25.355833+0000","last_undegraded":"2024-09-15T21:17:25.355833+0000","last_fullsized":"2024-09-15T21:17:25.355833+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17081,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-15T21:17:36.454103+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:36.454103+0000","last_peered":"2024-09-15T21:17:36.454103+0000","last_clean":"2024-09-15T21:17:36.454103+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:36.454103+0000","last_undegraded":"2024-09-15T21:17:36.454103+0000","last_fullsized":"2024-09-15T21:17:36.454103+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12508,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17239,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353047+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:25.353047+0000","last_peered":"2024-09-15T21:17:25.353047+0000","last_clean":"2024-09-15T21:17:25.353047+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:25.353047+0000","last_undegraded":"2024-09-15T21:17:25.353047+0000","last_fullsized":"2024-09-15T21:17:25.353047+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12541,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.352811+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:25.352811+0000","last_peered":"2024-09-15T21:17:25.352811+0000","last_clean":"2024-09-15T21:17:25.352811+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:25.352811+0000","last_undegraded":"2024-09-15T21:17:25.352811+0000","last_fullsized":"2024-09-15T21:17:25.352811+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12824,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353303+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:25.353303+0000","last_peered":"2024-09-15T21:17:25.353303+0000","last_clean":"2024-09-15T21:17:25.353303+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:25.353303+0000","last_undegraded":"2024-09-15T21:17:25.353303+0000","last_fullsized":"2024-09-15T21:17:25.353303+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13807,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-15T21:17:25.353509+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:25.353509+0000","last_peered":"2024-09-15T21:17:25.353509+0000","last_clean":"2024-09-15T21:17:25.353509+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:25.353509+0000","last_undegraded":"2024-09-15T21:17:25.353509+0000","last_fullsized":"2024-09-15T21:17:25.353509+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":40740,"num_objects":6,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2200,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":60277,"num_write_kb":201694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7045,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708288,"kb_used_data":492,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663552,"statfs":{"total":96636764160,"available":95911477248,"internally_reserved":0,"allocated":503808,"data_stored":223051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695292,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676548,"statfs":{"total":96636764160,"available":95924785152,"internally_reserved":0,"allocated":495616,"data_stored":219386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051747,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609664,"kb_used_data":496,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762176,"statfs":{"total":96636764160,"available":96012468224,"internally_reserved":0,"allocated":507904,"data_stored":231674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:41.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.1 is not active+clean 2024-09-15T21:17:41.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.1', 'version': "37'12657", 'reported_seq': 17399, 'reported_epoch': 59, 'state': 'peering', 'last_fresh': '2024-09-15T21:17:35.794409+0000', 'last_change': '2024-09-15T21:17:36.457811+0000', 'last_active': '2024-09-15T21:17:36.457811+0000', 'last_peered': '2024-09-15T21:17:35.794409+0000', 'last_clean': '2024-09-15T21:02:49.753923+0000', 'last_became_active': '2024-09-15T21:03:59.704275+0000', 'last_became_peered': '2024-09-15T21:03:59.704275+0000', 'last_unstale': '2024-09-15T21:17:35.794409+0000', 'last_undegraded': '2024-09-15T21:17:35.794409+0000', 'last_fullsized': '2024-09-15T21:03:59.043397+0000', 'mapping_epoch': 60, 'log_start': "31'2600", 'ondisk_log_start': "31'2600", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10057, 'log_dups_size': 0, 'ondisk_log_size': 10057, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 13580, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12655, 'num_write_kb': 50629, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2315, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 0, 2], 'acting': [3, 1, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:41.855 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:41.855 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 12837, 'reported_epoch': 60, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:36.627009+0000', 'last_change': '2024-09-15T21:17:35.795666+0000', 'last_active': '2024-09-15T21:17:36.627009+0000', 'last_peered': '2024-09-15T21:17:36.627009+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:36.627009+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2200, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 138, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -2196}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:41.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:42.229 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:42.229 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2200/24 objects degraded (9166.667%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":927,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"peering","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":6,"data_bytes":40740,"bytes_used":2792493056,"bytes_avail":383754563584,"bytes_total":386547056640,"inactive_pgs_ratio":0.0625,"degraded_objects":2200,"degraded_total":24,"degraded_ratio":91.666666666666686,"recovering_objects_per_sec":140,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1124,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:42.242 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:42.243 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:42.551 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:42.551 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:42.564 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":516,"stamp":"2024-09-15T21:17:42.237429+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735396,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374751964,"statfs":{"total":386547056640,"available":383746011136,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12655,"num_write_kb":-50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001241"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17654,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460351+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460351+0000","last_peered":"2024-09-15T21:17:37.460351+0000","last_clean":"2024-09-15T21:17:37.460351+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460351+0000","last_undegraded":"2024-09-15T21:17:37.460351+0000","last_fullsized":"2024-09-15T21:17:37.460351+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":14587,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:41.625855+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:41.625855+0000","last_peered":"2024-09-15T21:17:41.625855+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:41.625855+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617988,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753852,"statfs":{"total":96636764160,"available":96003944448,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247986,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:45.566 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:45.878 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:45.878 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:45.891 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-15T21:17:44.237745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735396,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374751964,"statfs":{"total":386547056640,"available":383746011136,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12655,"num_write_kb":-50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001198"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17655,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460403+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460403+0000","last_peered":"2024-09-15T21:17:37.460403+0000","last_clean":"2024-09-15T21:17:37.460403+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460403+0000","last_undegraded":"2024-09-15T21:17:37.460403+0000","last_fullsized":"2024-09-15T21:17:37.460403+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":14587,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:41.625855+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:41.625855+0000","last_peered":"2024-09-15T21:17:41.625855+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:41.625855+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617988,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753852,"statfs":{"total":96636764160,"available":96003944448,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:45.892 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:46.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:46.203 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:46.216 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-15T21:17:44.237745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2735396,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2733297,"kb_avail":374751964,"statfs":{"total":386547056640,"available":383746011136,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2798896690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12655,"num_write_kb":-50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001198"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17655,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460403+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460403+0000","last_peered":"2024-09-15T21:17:37.460403+0000","last_clean":"2024-09-15T21:17:37.460403+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460403+0000","last_undegraded":"2024-09-15T21:17:37.460403+0000","last_fullsized":"2024-09-15T21:17:37.460403+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":14587,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:41.625855+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:41.625855+0000","last_peered":"2024-09-15T21:17:41.625855+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:41.625855+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051748,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":617988,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":617467,"kb_avail":93753852,"statfs":{"total":96636764160,"available":96003944448,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":632286927},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:46.217 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:46.217 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 14587, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:41.625855+0000', 'last_change': '2024-09-15T21:17:35.795666+0000', 'last_active': '2024-09-15T21:17:41.625855+0000', 'last_peered': '2024-09-15T21:17:41.625855+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:41.625855+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1326, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1012, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -1322}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:46.217 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:46.616 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:46.616 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1326/16 objects degraded (8287.500%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":932,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":61,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":27160,"bytes_used":2792525824,"bytes_avail":383754530816,"bytes_total":386547056640,"degraded_objects":1326,"degraded_total":16,"degraded_ratio":82.875},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:46.630 INFO:tasks.ec_lost_unfound.ceph_manager:no progress seen, keeping timeout for now 2024-09-15T21:17:46.630 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:46.935 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:46.935 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:46.947 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-15T21:17:46.238040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727076,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760284,"statfs":{"total":386547056640,"available":383754530816,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13580,"num_objects":-2,"num_object_clones":0,"num_object_copies":-8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1998,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12655,"num_write_kb":-50629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-317,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.786792"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17655,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460403+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:37.460403+0000","last_peered":"2024-09-15T21:17:37.460403+0000","last_clean":"2024-09-15T21:17:37.460403+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:37.460403+0000","last_undegraded":"2024-09-15T21:17:37.460403+0000","last_fullsized":"2024-09-15T21:17:37.460403+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":14587,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:41.625855+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:41.625855+0000","last_peered":"2024-09-15T21:17:41.625855+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:41.625855+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-1322}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609668,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762172,"statfs":{"total":96636764160,"available":96012464128,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247987,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:49.952 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:50.247 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:50.247 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:50.259 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-15T21:17:50.238727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727076,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760284,"statfs":{"total":386547056640,"available":383754530816,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17657,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T21:17:48.498193+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:48.498193+0000","last_peered":"2024-09-15T21:17:48.498193+0000","last_clean":"2024-09-15T21:17:48.498193+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:48.498193+0000","last_undegraded":"2024-09-15T21:17:48.498193+0000","last_fullsized":"2024-09-15T21:17:48.498193+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":16657,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:46.627557+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:46.627557+0000","last_peered":"2024-09-15T21:17:46.627557+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:46.627557+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609668,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762172,"statfs":{"total":96636764160,"available":96012464128,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:50.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:50.557 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:50.557 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:50.569 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-15T21:17:50.238727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727076,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760284,"statfs":{"total":386547056640,"available":383754530816,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17657,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T21:17:48.498193+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:48.498193+0000","last_peered":"2024-09-15T21:17:48.498193+0000","last_clean":"2024-09-15T21:17:48.498193+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:48.498193+0000","last_undegraded":"2024-09-15T21:17:48.498193+0000","last_fullsized":"2024-09-15T21:17:48.498193+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":16657,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:46.627557+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:46.627557+0000","last_peered":"2024-09-15T21:17:46.627557+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:46.627557+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609668,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762172,"statfs":{"total":96636764160,"available":96012464128,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:50.570 INFO:tasks.ec_lost_unfound.ceph_manager:PG 1.4 is not active+clean 2024-09-15T21:17:50.570 INFO:tasks.ec_lost_unfound.ceph_manager:{'pgid': '1.4', 'version': "37'12448", 'reported_seq': 16657, 'reported_epoch': 61, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-15T21:17:46.627557+0000', 'last_change': '2024-09-15T21:17:35.795666+0000', 'last_active': '2024-09-15T21:17:46.627557+0000', 'last_peered': '2024-09-15T21:17:46.627557+0000', 'last_clean': '2024-09-15T21:02:50.365308+0000', 'last_became_active': '2024-09-15T21:03:59.704495+0000', 'last_became_peered': '2024-09-15T21:03:59.704495+0000', 'last_unstale': '2024-09-15T21:17:46.627557+0000', 'last_undegraded': '2024-09-15T21:03:59.130293+0000', 'last_fullsized': '2024-09-15T21:03:59.042911+0000', 'mapping_epoch': 30, 'log_start': "31'2400", 'ondisk_log_start': "31'2400", 'created': 13, 'last_epoch_clean': 15, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'last_clean_scrub_stamp': '2024-09-15T21:02:38.968547+0000', 'objects_scrubbed': 0, 'log_size': 10048, 'log_dups_size': 0, 'ondisk_log_size': 10048, 'stats_invalid': True, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 27160, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 16, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 291, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 12445, 'num_write_kb': 49793, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2047, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0, 2, 1], 'acting': [3, 0, 2, 2147483647], 'avail_no_missing': ['3(0)', '0(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': -287}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-15T21:17:50.570 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-15T21:17:50.918 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:50.918 INFO:teuthology.orchestra.run.smithi181.stdout:{"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 291/16 objects degraded (1818.750%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":936,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":63,"num_osds":4,"num_up_osds":4,"osd_up_since":1726434237,"num_in_osds":4,"osd_in_since":1726434136,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":15},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":16,"num_pools":1,"num_objects":4,"data_bytes":27160,"bytes_used":2792525824,"bytes_avail":383754530816,"bytes_total":386547056640,"degraded_objects":291,"degraded_total":16,"degraded_ratio":18.1875,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-15T21:02:14:250322+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-09-15T21:04:57.932863+0000","services":{}},"progress_events":{}} 2024-09-15T21:17:50.931 INFO:tasks.ec_lost_unfound.ceph_manager:making progress, resetting timeout 2024-09-15T21:17:50.932 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:51.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:51.214 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:51.226 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":522,"stamp":"2024-09-15T21:17:50.238727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2727076,"kb_used_data":2000,"kb_used_omap":14,"kb_used_meta":2724977,"kb_avail":374760284,"statfs":{"total":386547056640,"available":383754530816,"internally_reserved":0,"allocated":2048000,"data_stored":906780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2790377010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1035,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1035,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001297"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17020,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460471+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:37.460471+0000","last_peered":"2024-09-15T21:17:37.460471+0000","last_clean":"2024-09-15T21:17:37.460471+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:37.460471+0000","last_undegraded":"2024-09-15T21:17:37.460471+0000","last_fullsized":"2024-09-15T21:17:37.460471+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17498,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465995+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:37.465995+0000","last_peered":"2024-09-15T21:17:37.465995+0000","last_clean":"2024-09-15T21:17:37.465995+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:37.465995+0000","last_undegraded":"2024-09-15T21:17:37.465995+0000","last_fullsized":"2024-09-15T21:17:37.465995+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17108,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456732+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:37.456732+0000","last_peered":"2024-09-15T21:17:37.456732+0000","last_clean":"2024-09-15T21:17:37.456732+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:37.456732+0000","last_undegraded":"2024-09-15T21:17:37.456732+0000","last_fullsized":"2024-09-15T21:17:37.456732+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12586,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465774+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:37.465774+0000","last_peered":"2024-09-15T21:17:37.465774+0000","last_clean":"2024-09-15T21:17:37.465774+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:37.465774+0000","last_undegraded":"2024-09-15T21:17:37.465774+0000","last_fullsized":"2024-09-15T21:17:37.465774+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17418,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:38.507689+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:38.507689+0000","last_peered":"2024-09-15T21:17:38.507689+0000","last_clean":"2024-09-15T21:17:38.507689+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:38.507689+0000","last_undegraded":"2024-09-15T21:17:38.507689+0000","last_fullsized":"2024-09-15T21:17:38.507689+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17123,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.461932+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:37.461932+0000","last_peered":"2024-09-15T21:17:37.461932+0000","last_clean":"2024-09-15T21:17:37.461932+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:37.461932+0000","last_undegraded":"2024-09-15T21:17:37.461932+0000","last_fullsized":"2024-09-15T21:17:37.461932+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17657,"reported_epoch":62,"state":"active+clean","last_fresh":"2024-09-15T21:17:48.498193+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:48.498193+0000","last_peered":"2024-09-15T21:17:48.498193+0000","last_clean":"2024-09-15T21:17:48.498193+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:48.498193+0000","last_undegraded":"2024-09-15T21:17:48.498193+0000","last_fullsized":"2024-09-15T21:17:48.498193+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12882,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466047+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:37.466047+0000","last_peered":"2024-09-15T21:17:37.466047+0000","last_clean":"2024-09-15T21:17:37.466047+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:37.466047+0000","last_undegraded":"2024-09-15T21:17:37.466047+0000","last_fullsized":"2024-09-15T21:17:37.466047+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":16657,"reported_epoch":61,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-15T21:17:46.627557+0000","last_change":"2024-09-15T21:17:35.795666+0000","last_active":"2024-09-15T21:17:46.627557+0000","last_peered":"2024-09-15T21:17:46.627557+0000","last_clean":"2024-09-15T21:02:50.365308+0000","last_became_active":"2024-09-15T21:03:59.704495+0000","last_became_peered":"2024-09-15T21:03:59.704495+0000","last_unstale":"2024-09-15T21:17:46.627557+0000","last_undegraded":"2024-09-15T21:03:59.130293+0000","last_fullsized":"2024-09-15T21:03:59.042911+0000","mapping_epoch":30,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12445,"num_write_kb":49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,2147483647],"avail_no_missing":["3(0)","0(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":-287}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17394,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.456849+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:37.456849+0000","last_peered":"2024-09-15T21:17:37.456849+0000","last_clean":"2024-09-15T21:17:37.456849+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:37.456849+0000","last_undegraded":"2024-09-15T21:17:37.456849+0000","last_fullsized":"2024-09-15T21:17:37.456849+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17084,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.460469+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:37.460469+0000","last_peered":"2024-09-15T21:17:37.460469+0000","last_clean":"2024-09-15T21:17:37.460469+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:37.460469+0000","last_undegraded":"2024-09-15T21:17:37.460469+0000","last_fullsized":"2024-09-15T21:17:37.460469+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12512,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466054+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:37.466054+0000","last_peered":"2024-09-15T21:17:37.466054+0000","last_clean":"2024-09-15T21:17:37.466054+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:37.466054+0000","last_undegraded":"2024-09-15T21:17:37.466054+0000","last_fullsized":"2024-09-15T21:17:37.466054+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17243,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465810+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:37.465810+0000","last_peered":"2024-09-15T21:17:37.465810+0000","last_clean":"2024-09-15T21:17:37.465810+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:37.465810+0000","last_undegraded":"2024-09-15T21:17:37.465810+0000","last_fullsized":"2024-09-15T21:17:37.465810+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12545,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.466218+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:37.466218+0000","last_peered":"2024-09-15T21:17:37.466218+0000","last_clean":"2024-09-15T21:17:37.466218+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:37.466218+0000","last_undegraded":"2024-09-15T21:17:37.466218+0000","last_fullsized":"2024-09-15T21:17:37.466218+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12828,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465773+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:37.465773+0000","last_peered":"2024-09-15T21:17:37.465773+0000","last_clean":"2024-09-15T21:17:37.465773+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:37.465773+0000","last_undegraded":"2024-09-15T21:17:37.465773+0000","last_fullsized":"2024-09-15T21:17:37.465773+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13812,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-15T21:17:37.465960+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:37.465960+0000","last_peered":"2024-09-15T21:17:37.465960+0000","last_clean":"2024-09-15T21:17:37.465960+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:37.465960+0000","last_undegraded":"2024-09-15T21:17:37.465960+0000","last_fullsized":"2024-09-15T21:17:37.465960+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":27160,"num_objects":4,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47622,"num_write_kb":151065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708300,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663540,"statfs":{"total":96636764160,"available":95911464960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695308,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676532,"statfs":{"total":96636764160,"available":95924768768,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051749,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":609668,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":609147,"kb_avail":93762172,"statfs":{"total":96636764160,"available":96012464128,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":623767247},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247988,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713800,"kb_used_data":500,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658040,"statfs":{"total":96636764160,"available":95905832960,"internally_reserved":0,"allocated":512000,"data_stored":226695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:54.228 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:54.526 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:54.527 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:54.539 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T21:17:54.539 INFO:tasks.ec_lost_unfound.ceph_manager:waiting for clean 2024-09-15T21:17:54.539 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:54.820 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:54.821 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:54.833 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:54.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:55.115 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:55.116 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:55.128 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:55.129 INFO:tasks.ec_lost_unfound.ceph_manager:clean! 2024-09-15T21:17:55.129 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T21:17:55.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T21:17:55.141 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:55.422 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:55.423 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:55.436 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:55.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:55.719 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:55.720 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:55.732 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:55.733 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T21:17:55.733 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:56.013 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:56.013 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:56.025 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2024-09-15T21:17:54.239544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1012,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001504"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17260,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434872+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434872+0000","last_peered":"2024-09-15T21:17:50.434872+0000","last_clean":"2024-09-15T21:17:50.434872+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434872+0000","last_undegraded":"2024-09-15T21:17:50.434872+0000","last_fullsized":"2024-09-15T21:17:50.434872+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149874,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051750,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:56.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T21:17:56.309 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:56.309 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":63,"fsid":"c31fa425-5a82-4117-b11e-63b4944cd2c3","created":"2024-09-15T21:02:14.252559+0000","modified":"2024-09-15T21:17:49.494603+0000","last_up_change":"2024-09-15T21:03:57.008004+0000","last_in_change":"2024-09-15T21:02:16.620369+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-09-15T21:02:37.977251+0000","flags":1,"flags_names":"hashpspool","type":3,"size":4,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"lost_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"92896e8a-721b-45df-b061-1807b46a1f81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":16,"up_from":23,"up_thru":49,"down_at":17,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6809","nonce":2996802086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6811","nonce":2996802086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6815","nonce":2996802086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2996802086},{"type":"v1","addr":"172.21.15.181:6813","nonce":2996802086}]},"public_addr":"172.21.15.181:6809/2996802086","cluster_addr":"172.21.15.181:6811/2996802086","heartbeat_back_addr":"172.21.15.181:6815/2996802086","heartbeat_front_addr":"172.21.15.181:6813/2996802086","state":["exists","up"]},{"osd":1,"uuid":"84736d86-cc39-4c9a-9154-48a566b2eb39","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":19,"up_from":29,"up_thru":54,"down_at":20,"lost_at":20,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6825","nonce":36577163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6827","nonce":36577163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6831","nonce":36577163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":36577163},{"type":"v1","addr":"172.21.15.181:6829","nonce":36577163}]},"public_addr":"172.21.15.181:6825/36577163","cluster_addr":"172.21.15.181:6827/36577163","heartbeat_back_addr":"172.21.15.181:6831/36577163","heartbeat_front_addr":"172.21.15.181:6829/36577163","state":["exists","up"]},{"osd":2,"uuid":"198fc7a7-b64a-4498-95bc-18995cfaa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6801","nonce":2018078012}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6803","nonce":2018078012}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6807","nonce":2018078012}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2018078012},{"type":"v1","addr":"172.21.15.181:6805","nonce":2018078012}]},"public_addr":"172.21.15.181:6801/2018078012","cluster_addr":"172.21.15.181:6803/2018078012","heartbeat_back_addr":"172.21.15.181:6807/2018078012","heartbeat_front_addr":"172.21.15.181:6805/2018078012","state":["exists","up"]},{"osd":3,"uuid":"f42cf73c-6ff4-4301-b660-9e56cc9bed0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":17,"up_from":26,"up_thru":62,"down_at":18,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6817","nonce":2345571741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6819","nonce":2345571741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6823","nonce":2345571741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":2345571741},{"type":"v1","addr":"172.21.15.181:6821","nonce":2345571741}]},"public_addr":"172.21.15.181:6817/2345571741","cluster_addr":"172.21.15.181:6819/2345571741","heartbeat_back_addr":"172.21.15.181:6823/2345571741","heartbeat_front_addr":"172.21.15.181:6821/2345571741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.482742+0000","dead_epoch":17},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.186609+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.138541+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T21:02:23.649252+0000","dead_epoch":18}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"lost_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T21:17:57.324 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T21:17:57.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T21:17:57.454 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:17:57.454 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T21:17:57.454 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:17:57.464 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T21:17:57.749 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T21:17:57.762 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T21:17:57.762 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T21:17:57.883 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:17:57.883 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-09-15T21:17:57.883 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:17:57.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T21:17:58.178 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T21:17:58.191 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T21:17:58.191 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T21:17:58.313 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:17:58.313 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T21:17:58.313 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:17:58.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T21:17:58.611 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T21:17:58.624 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T21:17:58.624 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T21:17:58.746 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2024-09-15T21:17:58.746 INFO:teuthology.orchestra.run.smithi181.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T21:17:58.746 INFO:teuthology.orchestra.run.smithi181.stdout:} 2024-09-15T21:17:58.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T21:17:59.041 INFO:teuthology.orchestra.run.smithi181.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T21:17:59.054 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:17:59.340 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:17:59.341 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:17:59.354 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-09-15T21:17:58.240204+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731308,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756052,"statfs":{"total":386547056640,"available":383750197248,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20370,"num_objects":-3,"num_object_clones":0,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12445,"num_write_kb":-49793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2047,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.743875"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17024,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502858+0000","last_change":"2024-09-15T21:15:36.376957+0000","last_active":"2024-09-15T21:17:49.502858+0000","last_peered":"2024-09-15T21:17:49.502858+0000","last_clean":"2024-09-15T21:17:49.502858+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:17:49.502858+0000","last_undegraded":"2024-09-15T21:17:49.502858+0000","last_fullsized":"2024-09-15T21:17:49.502858+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:25.971071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17502,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504795+0000","last_change":"2024-09-15T21:10:48.024403+0000","last_active":"2024-09-15T21:17:49.504795+0000","last_peered":"2024-09-15T21:17:49.504795+0000","last_clean":"2024-09-15T21:17:49.504795+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:17:49.504795+0000","last_undegraded":"2024-09-15T21:17:49.504795+0000","last_fullsized":"2024-09-15T21:17:49.504795+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"35'9371","last_scrub_stamp":"2024-09-15T21:10:48.023986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:10:48.023986+0000","objects_scrubbed":3216,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:27:11.983231+0000","scrub_duration":6228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17112,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501991+0000","last_change":"2024-09-15T21:17:14.072679+0000","last_active":"2024-09-15T21:17:49.501991+0000","last_peered":"2024-09-15T21:17:49.501991+0000","last_clean":"2024-09-15T21:17:49.501991+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:49.501991+0000","last_undegraded":"2024-09-15T21:17:49.501991+0000","last_fullsized":"2024-09-15T21:17:49.501991+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:14.072611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:14.072611+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:34:13.432039+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12590,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504520+0000","last_change":"2024-09-15T21:16:11.762034+0000","last_active":"2024-09-15T21:17:49.504520+0000","last_peered":"2024-09-15T21:17:49.504520+0000","last_clean":"2024-09-15T21:17:49.504520+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:49.504520+0000","last_undegraded":"2024-09-15T21:17:49.504520+0000","last_fullsized":"2024-09-15T21:17:49.504520+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:16:11.761963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:11.761963+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:41:05.142435+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17422,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502943+0000","last_change":"2024-09-15T21:17:38.507689+0000","last_active":"2024-09-15T21:17:49.502943+0000","last_peered":"2024-09-15T21:17:49.502943+0000","last_clean":"2024-09-15T21:17:49.502943+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:17:49.502943+0000","last_undegraded":"2024-09-15T21:17:49.502943+0000","last_fullsized":"2024-09-15T21:17:49.502943+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:17:38.507618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:38.507618+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:47:19.570231+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17127,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502379+0000","last_change":"2024-09-15T21:17:25.437548+0000","last_active":"2024-09-15T21:17:49.502379+0000","last_peered":"2024-09-15T21:17:49.502379+0000","last_clean":"2024-09-15T21:17:49.502379+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:17:49.502379+0000","last_undegraded":"2024-09-15T21:17:49.502379+0000","last_fullsized":"2024-09-15T21:17:49.502379+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:02:38.968547+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:26:52.621038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17659,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502615+0000","last_change":"2024-09-15T21:15:49.080697+0000","last_active":"2024-09-15T21:17:49.502615+0000","last_peered":"2024-09-15T21:17:49.502615+0000","last_clean":"2024-09-15T21:17:49.502615+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:49.502615+0000","last_undegraded":"2024-09-15T21:17:49.502615+0000","last_fullsized":"2024-09-15T21:17:49.502615+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:15:49.080606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:49.080606+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:54:27.738424+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12886,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504700+0000","last_change":"2024-09-15T21:16:36.673452+0000","last_active":"2024-09-15T21:17:49.504700+0000","last_peered":"2024-09-15T21:17:49.504700+0000","last_clean":"2024-09-15T21:17:49.504700+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:17:49.504700+0000","last_undegraded":"2024-09-15T21:17:49.504700+0000","last_fullsized":"2024-09-15T21:17:49.504700+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:16:36.673347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:36.673347+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:49:37.121949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17261,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:50.434966+0000","last_change":"2024-09-15T21:17:50.434872+0000","last_active":"2024-09-15T21:17:50.434966+0000","last_peered":"2024-09-15T21:17:50.434966+0000","last_clean":"2024-09-15T21:17:50.434966+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:50.434966+0000","last_undegraded":"2024-09-15T21:17:50.434966+0000","last_fullsized":"2024-09-15T21:17:50.434966+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:50.434303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:50.434303+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:21:10.804981+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17398,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.501980+0000","last_change":"2024-09-15T21:16:48.977744+0000","last_active":"2024-09-15T21:17:49.501980+0000","last_peered":"2024-09-15T21:17:49.501980+0000","last_clean":"2024-09-15T21:17:49.501980+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:49.501980+0000","last_undegraded":"2024-09-15T21:17:49.501980+0000","last_fullsized":"2024-09-15T21:17:49.501980+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:16:48.977663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:48.977663+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:57:04.756904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17088,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.502784+0000","last_change":"2024-09-15T21:07:08.972188+0000","last_active":"2024-09-15T21:17:49.502784+0000","last_peered":"2024-09-15T21:17:49.502784+0000","last_clean":"2024-09-15T21:17:49.502784+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:17:49.502784+0000","last_undegraded":"2024-09-15T21:17:49.502784+0000","last_fullsized":"2024-09-15T21:17:49.502784+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'5935","last_scrub_stamp":"2024-09-15T21:07:08.971798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:07:08.971798+0000","objects_scrubbed":5881,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":8,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:52:13.587549+0000","scrub_duration":7363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12516,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504808+0000","last_change":"2024-09-15T21:17:00.883953+0000","last_active":"2024-09-15T21:17:49.504808+0000","last_peered":"2024-09-15T21:17:49.504808+0000","last_clean":"2024-09-15T21:17:49.504808+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:49.504808+0000","last_undegraded":"2024-09-15T21:17:49.504808+0000","last_fullsized":"2024-09-15T21:17:49.504808+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:00.883844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:17:00.883844+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:04:47.383598+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17247,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504535+0000","last_change":"2024-09-15T21:14:09.598529+0000","last_active":"2024-09-15T21:17:49.504535+0000","last_peered":"2024-09-15T21:17:49.504535+0000","last_clean":"2024-09-15T21:17:49.504535+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:17:49.504535+0000","last_undegraded":"2024-09-15T21:17:49.504535+0000","last_fullsized":"2024-09-15T21:17:49.504535+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'11742","last_scrub_stamp":"2024-09-15T21:14:09.598393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:14:09.598393+0000","objects_scrubbed":762,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:25:59.661113+0000","scrub_duration":1154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12549,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504579+0000","last_change":"2024-09-15T21:16:23.609955+0000","last_active":"2024-09-15T21:17:49.504579+0000","last_peered":"2024-09-15T21:17:49.504579+0000","last_clean":"2024-09-15T21:17:49.504579+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:17:49.504579+0000","last_undegraded":"2024-09-15T21:17:49.504579+0000","last_fullsized":"2024-09-15T21:17:49.504579+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:16:23.609892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:16:23.609892+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:27:05.931783+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12832,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504671+0000","last_change":"2024-09-15T21:15:59.721885+0000","last_active":"2024-09-15T21:17:49.504671+0000","last_peered":"2024-09-15T21:17:49.504671+0000","last_clean":"2024-09-15T21:17:49.504671+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:17:49.504671+0000","last_undegraded":"2024-09-15T21:17:49.504671+0000","last_fullsized":"2024-09-15T21:17:49.504671+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:15:59.721806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:59.721806+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:10:04.481938+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13816,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:49.504665+0000","last_change":"2024-09-15T21:15:25.660461+0000","last_active":"2024-09-15T21:17:49.504665+0000","last_peered":"2024-09-15T21:17:49.504665+0000","last_clean":"2024-09-15T21:17:49.504665+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:17:49.504665+0000","last_undegraded":"2024-09-15T21:17:49.504665+0000","last_fullsized":"2024-09-15T21:17:49.504665+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:15:25.660383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T21:02:38.968547+0000","last_clean_scrub_stamp":"2024-09-15T21:15:25.660383+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:49:31.605224+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708316,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663524,"statfs":{"total":96636764160,"available":95911448576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695324,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676516,"statfs":{"total":96636764160,"available":95924752384,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051751,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613852,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757988,"statfs":{"total":96636764160,"available":96008179712,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247989,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713816,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658024,"statfs":{"total":96636764160,"available":95905816576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-09-15T21:02:38.968547+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=2, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-09-15T21:10:48.023986+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=10, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-09-15T21:17:14.072611+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=14, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T21:16:11.761963+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=16, tm_sec=11, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-15T21:17:38.507618+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-09-15T21:02:38.968547+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=2, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.361 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-15T21:15:49.080606+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=15, tm_sec=49, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-15T21:16:36.673347+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=16, tm_sec=36, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-15T21:17:50.434303+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=50, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-15T21:16:48.977663+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=16, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-15T21:07:08.971798+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=7, tm_sec=8, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-15T21:17:00.883844+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=0, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-09-15T21:14:09.598393+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=14, tm_sec=9, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-09-15T21:16:23.609892+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=16, tm_sec=23, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-09-15T21:15:59.721806+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=15, tm_sec=59, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.362 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-09-15T21:15:25.660383+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=15, tm_sec=25, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=21, tm_min=17, tm_sec=56, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T21:17:59.363 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T21:18:19.364 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T21:18:19.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-15T21:18:19.664 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-15T21:18:19.675 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-09-15T21:18:18.244292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":2731280,"kb_used_data":2064,"kb_used_omap":14,"kb_used_meta":2729137,"kb_avail":374756080,"statfs":{"total":386547056640,"available":383750225920,"internally_reserved":0,"allocated":2113536,"data_stored":935720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14798,"internal_metadata":2794636850},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002612"},"pg_stats":[{"pgid":"1.f","version":"37'12352","reported_seq":17032,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:02.309140+0000","last_change":"2024-09-15T21:18:02.309140+0000","last_active":"2024-09-15T21:18:02.309140+0000","last_peered":"2024-09-15T21:18:02.309140+0000","last_clean":"2024-09-15T21:18:02.309140+0000","last_became_active":"2024-09-15T21:15:36.299297+0000","last_became_peered":"2024-09-15T21:15:36.299297+0000","last_unstale":"2024-09-15T21:18:02.309140+0000","last_undegraded":"2024-09-15T21:18:02.309140+0000","last_fullsized":"2024-09-15T21:18:02.309140+0000","mapping_epoch":40,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12352","last_scrub_stamp":"2024-09-15T21:18:02.309032+0000","last_deep_scrub":"37'12352","last_deep_scrub_stamp":"2024-09-15T21:18:02.309032+0000","last_clean_scrub_stamp":"2024-09-15T21:18:02.309032+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:01:18.552780+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2276,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"37'12550","reported_seq":17510,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:05.753606+0000","last_change":"2024-09-15T21:18:05.753606+0000","last_active":"2024-09-15T21:18:05.753606+0000","last_peered":"2024-09-15T21:18:05.753606+0000","last_clean":"2024-09-15T21:18:05.753606+0000","last_became_active":"2024-09-15T21:10:41.499163+0000","last_became_peered":"2024-09-15T21:10:41.499163+0000","last_unstale":"2024-09-15T21:18:05.753606+0000","last_undegraded":"2024-09-15T21:18:05.753606+0000","last_fullsized":"2024-09-15T21:18:05.753606+0000","mapping_epoch":34,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12550","last_scrub_stamp":"2024-09-15T21:18:05.753498+0000","last_deep_scrub":"37'12550","last_deep_scrub_stamp":"2024-09-15T21:18:05.753498+0000","last_clean_scrub_stamp":"2024-09-15T21:18:05.753498+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:53:00.667144+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3,2],"acting":[1,0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"37'12432","reported_seq":17120,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:59.768422+0000","last_change":"2024-09-15T21:17:59.768422+0000","last_active":"2024-09-15T21:17:59.768422+0000","last_peered":"2024-09-15T21:17:59.768422+0000","last_clean":"2024-09-15T21:17:59.768422+0000","last_became_active":"2024-09-15T21:17:13.392336+0000","last_became_peered":"2024-09-15T21:17:13.392336+0000","last_unstale":"2024-09-15T21:17:59.768422+0000","last_undegraded":"2024-09-15T21:17:59.768422+0000","last_fullsized":"2024-09-15T21:17:59.768422+0000","mapping_epoch":56,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12432","last_scrub_stamp":"2024-09-15T21:17:59.768336+0000","last_deep_scrub":"37'12432","last_deep_scrub_stamp":"2024-09-15T21:17:59.768336+0000","last_clean_scrub_stamp":"2024-09-15T21:17:59.768336+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:53:11.653527+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,0],"acting":[2,1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"37'12454","reported_seq":12599,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:58.739460+0000","last_change":"2024-09-15T21:17:58.739406+0000","last_active":"2024-09-15T21:17:58.739460+0000","last_peered":"2024-09-15T21:17:58.739460+0000","last_clean":"2024-09-15T21:17:58.739460+0000","last_became_active":"2024-09-15T21:16:11.393133+0000","last_became_peered":"2024-09-15T21:16:11.393133+0000","last_unstale":"2024-09-15T21:17:58.739460+0000","last_undegraded":"2024-09-15T21:17:58.739460+0000","last_fullsized":"2024-09-15T21:17:58.739460+0000","mapping_epoch":46,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12454","last_scrub_stamp":"2024-09-15T21:17:58.739312+0000","last_deep_scrub":"37'12454","last_deep_scrub_stamp":"2024-09-15T21:17:58.739312+0000","last_clean_scrub_stamp":"2024-09-15T21:17:58.739312+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:11:21.678574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"37'12657","reported_seq":17430,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:00.340005+0000","last_change":"2024-09-15T21:18:00.340005+0000","last_active":"2024-09-15T21:18:00.340005+0000","last_peered":"2024-09-15T21:18:00.340005+0000","last_clean":"2024-09-15T21:18:00.340005+0000","last_became_active":"2024-09-15T21:17:37.467076+0000","last_became_peered":"2024-09-15T21:17:37.467076+0000","last_unstale":"2024-09-15T21:18:00.340005+0000","last_undegraded":"2024-09-15T21:18:00.340005+0000","last_fullsized":"2024-09-15T21:18:00.340005+0000","mapping_epoch":60,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12657","last_scrub_stamp":"2024-09-15T21:18:00.339891+0000","last_deep_scrub":"37'12657","last_deep_scrub_stamp":"2024-09-15T21:18:00.339891+0000","last_clean_scrub_stamp":"2024-09-15T21:18:00.339891+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:17:14.979097+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0,2],"acting":[3,1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"37'12376","reported_seq":17135,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:03.268802+0000","last_change":"2024-09-15T21:18:03.268802+0000","last_active":"2024-09-15T21:18:03.268802+0000","last_peered":"2024-09-15T21:18:03.268802+0000","last_clean":"2024-09-15T21:18:03.268802+0000","last_became_active":"2024-09-15T21:17:25.356750+0000","last_became_peered":"2024-09-15T21:17:25.356750+0000","last_unstale":"2024-09-15T21:18:03.268802+0000","last_undegraded":"2024-09-15T21:18:03.268802+0000","last_fullsized":"2024-09-15T21:18:03.268802+0000","mapping_epoch":58,"log_start":"28'2300","ondisk_log_start":"28'2300","created":13,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12376","last_scrub_stamp":"2024-09-15T21:18:03.268700+0000","last_deep_scrub":"37'12376","last_deep_scrub_stamp":"2024-09-15T21:18:03.268700+0000","last_clean_scrub_stamp":"2024-09-15T21:18:03.268700+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:34:59.084340+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12376,"num_write_kb":49504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2316,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"37'12770","reported_seq":17667,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:58.338292+0000","last_change":"2024-09-15T21:17:58.338292+0000","last_active":"2024-09-15T21:17:58.338292+0000","last_peered":"2024-09-15T21:17:58.338292+0000","last_clean":"2024-09-15T21:17:58.338292+0000","last_became_active":"2024-09-15T21:15:48.335635+0000","last_became_peered":"2024-09-15T21:15:48.335635+0000","last_unstale":"2024-09-15T21:17:58.338292+0000","last_undegraded":"2024-09-15T21:17:58.338292+0000","last_fullsized":"2024-09-15T21:17:58.338292+0000","mapping_epoch":42,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12770","last_scrub_stamp":"2024-09-15T21:17:58.338211+0000","last_deep_scrub":"37'12770","last_deep_scrub_stamp":"2024-09-15T21:17:58.338211+0000","last_clean_scrub_stamp":"2024-09-15T21:17:58.338211+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:05:19.059016+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1,2],"acting":[0,3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"37'12759","reported_seq":12894,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:00.761150+0000","last_change":"2024-09-15T21:18:00.761150+0000","last_active":"2024-09-15T21:18:00.761150+0000","last_peered":"2024-09-15T21:18:00.761150+0000","last_clean":"2024-09-15T21:18:00.761150+0000","last_became_active":"2024-09-15T21:16:36.477373+0000","last_became_peered":"2024-09-15T21:16:36.477373+0000","last_unstale":"2024-09-15T21:18:00.761150+0000","last_undegraded":"2024-09-15T21:18:00.761150+0000","last_fullsized":"2024-09-15T21:18:00.761150+0000","mapping_epoch":50,"log_start":"31'2700","ondisk_log_start":"31'2700","created":13,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12759","last_scrub_stamp":"2024-09-15T21:18:00.761047+0000","last_deep_scrub":"37'12759","last_deep_scrub_stamp":"2024-09-15T21:18:00.761047+0000","last_clean_scrub_stamp":"2024-09-15T21:18:00.761047+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:47:05.746729+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"37'12448","reported_seq":17269,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:59.344421+0000","last_change":"2024-09-15T21:17:59.344421+0000","last_active":"2024-09-15T21:17:59.344421+0000","last_peered":"2024-09-15T21:17:59.344421+0000","last_clean":"2024-09-15T21:17:59.344421+0000","last_became_active":"2024-09-15T21:17:49.505598+0000","last_became_peered":"2024-09-15T21:17:49.505598+0000","last_unstale":"2024-09-15T21:17:59.344421+0000","last_undegraded":"2024-09-15T21:17:59.344421+0000","last_fullsized":"2024-09-15T21:17:59.344421+0000","mapping_epoch":62,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12448","last_scrub_stamp":"2024-09-15T21:17:59.344342+0000","last_deep_scrub":"37'12448","last_deep_scrub_stamp":"2024-09-15T21:17:59.344342+0000","last_clean_scrub_stamp":"2024-09-15T21:17:59.344342+0000","objects_scrubbed":1,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:31.965921+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2,1],"acting":[3,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"37'12560","reported_seq":17406,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:58.780302+0000","last_change":"2024-09-15T21:17:58.780302+0000","last_active":"2024-09-15T21:17:58.780302+0000","last_peered":"2024-09-15T21:17:58.780302+0000","last_clean":"2024-09-15T21:17:58.780302+0000","last_became_active":"2024-09-15T21:16:48.518043+0000","last_became_peered":"2024-09-15T21:16:48.518043+0000","last_unstale":"2024-09-15T21:17:58.780302+0000","last_undegraded":"2024-09-15T21:17:58.780302+0000","last_fullsized":"2024-09-15T21:17:58.780302+0000","mapping_epoch":52,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12560","last_scrub_stamp":"2024-09-15T21:17:58.780195+0000","last_deep_scrub":"37'12560","last_deep_scrub_stamp":"2024-09-15T21:17:58.780195+0000","last_clean_scrub_stamp":"2024-09-15T21:17:58.780195+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:00:01.354871+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3,1],"acting":[2,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"37'12360","reported_seq":17096,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:01.322623+0000","last_change":"2024-09-15T21:18:01.322623+0000","last_active":"2024-09-15T21:18:01.322623+0000","last_peered":"2024-09-15T21:18:01.322623+0000","last_clean":"2024-09-15T21:18:01.322623+0000","last_became_active":"2024-09-15T21:07:01.480618+0000","last_became_peered":"2024-09-15T21:07:01.480618+0000","last_unstale":"2024-09-15T21:18:01.322623+0000","last_undegraded":"2024-09-15T21:18:01.322623+0000","last_fullsized":"2024-09-15T21:18:01.322623+0000","mapping_epoch":32,"log_start":"31'2300","ondisk_log_start":"31'2300","created":13,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12360","last_scrub_stamp":"2024-09-15T21:18:01.322462+0000","last_deep_scrub":"37'12360","last_deep_scrub_stamp":"2024-09-15T21:18:01.322462+0000","last_clean_scrub_stamp":"2024-09-15T21:18:01.322462+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:45:37.341034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6471,"num_write_kb":2360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1,2],"acting":[3,0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"37'12426","reported_seq":12524,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:17:59.755125+0000","last_change":"2024-09-15T21:17:59.755125+0000","last_active":"2024-09-15T21:17:59.755125+0000","last_peered":"2024-09-15T21:17:59.755125+0000","last_clean":"2024-09-15T21:17:59.755125+0000","last_became_active":"2024-09-15T21:17:00.352741+0000","last_became_peered":"2024-09-15T21:17:00.352741+0000","last_unstale":"2024-09-15T21:17:59.755125+0000","last_undegraded":"2024-09-15T21:17:59.755125+0000","last_fullsized":"2024-09-15T21:17:59.755125+0000","mapping_epoch":54,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12426","last_scrub_stamp":"2024-09-15T21:17:59.755020+0000","last_deep_scrub":"37'12426","last_deep_scrub_stamp":"2024-09-15T21:17:59.755020+0000","last_clean_scrub_stamp":"2024-09-15T21:17:59.755020+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:49:54.459849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"37'12500","reported_seq":17255,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:03.747966+0000","last_change":"2024-09-15T21:18:03.747966+0000","last_active":"2024-09-15T21:18:03.747966+0000","last_peered":"2024-09-15T21:18:03.747966+0000","last_clean":"2024-09-15T21:18:03.747966+0000","last_became_active":"2024-09-15T21:14:07.529588+0000","last_became_peered":"2024-09-15T21:14:07.529588+0000","last_unstale":"2024-09-15T21:18:03.747966+0000","last_undegraded":"2024-09-15T21:18:03.747966+0000","last_fullsized":"2024-09-15T21:18:03.747966+0000","mapping_epoch":36,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12500","last_scrub_stamp":"2024-09-15T21:18:03.747839+0000","last_deep_scrub":"37'12500","last_deep_scrub_stamp":"2024-09-15T21:18:03.747839+0000","last_clean_scrub_stamp":"2024-09-15T21:18:03.747839+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:57:16.476152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":762,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"37'12430","reported_seq":12557,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:01.781806+0000","last_change":"2024-09-15T21:18:01.781806+0000","last_active":"2024-09-15T21:18:01.781806+0000","last_peered":"2024-09-15T21:18:01.781806+0000","last_clean":"2024-09-15T21:18:01.781806+0000","last_became_active":"2024-09-15T21:16:23.430070+0000","last_became_peered":"2024-09-15T21:16:23.430070+0000","last_unstale":"2024-09-15T21:18:01.781806+0000","last_undegraded":"2024-09-15T21:18:01.781806+0000","last_fullsized":"2024-09-15T21:18:01.781806+0000","mapping_epoch":48,"log_start":"31'2400","ondisk_log_start":"31'2400","created":13,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12430","last_scrub_stamp":"2024-09-15T21:18:01.781700+0000","last_deep_scrub":"37'12430","last_deep_scrub_stamp":"2024-09-15T21:18:01.781700+0000","last_clean_scrub_stamp":"2024-09-15T21:18:01.781700+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:07:41.326959+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2,0],"acting":[1,3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"37'12679","reported_seq":12841,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:02.761210+0000","last_change":"2024-09-15T21:18:02.761110+0000","last_active":"2024-09-15T21:18:02.761210+0000","last_peered":"2024-09-15T21:18:02.761210+0000","last_clean":"2024-09-15T21:18:02.761210+0000","last_became_active":"2024-09-15T21:15:59.357463+0000","last_became_peered":"2024-09-15T21:15:59.357463+0000","last_unstale":"2024-09-15T21:18:02.761210+0000","last_undegraded":"2024-09-15T21:18:02.761210+0000","last_fullsized":"2024-09-15T21:18:02.761210+0000","mapping_epoch":44,"log_start":"31'2600","ondisk_log_start":"31'2600","created":13,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12679","last_scrub_stamp":"2024-09-15T21:18:02.760952+0000","last_deep_scrub":"37'12679","last_deep_scrub_stamp":"2024-09-15T21:18:02.760952+0000","last_clean_scrub_stamp":"2024-09-15T21:18:02.760952+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:55:13.576233+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,0],"acting":[1,2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.b","version":"37'12547","reported_seq":13824,"reported_epoch":63,"state":"active+clean","last_fresh":"2024-09-15T21:18:04.752253+0000","last_change":"2024-09-15T21:18:04.752253+0000","last_active":"2024-09-15T21:18:04.752253+0000","last_peered":"2024-09-15T21:18:04.752253+0000","last_clean":"2024-09-15T21:18:04.752253+0000","last_became_active":"2024-09-15T21:15:24.630738+0000","last_became_peered":"2024-09-15T21:15:24.630738+0000","last_unstale":"2024-09-15T21:18:04.752253+0000","last_undegraded":"2024-09-15T21:18:04.752253+0000","last_fullsized":"2024-09-15T21:18:04.752253+0000","mapping_epoch":38,"log_start":"31'2500","ondisk_log_start":"31'2500","created":13,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"37'12547","last_scrub_stamp":"2024-09-15T21:18:04.752150+0000","last_deep_scrub":"37'12547","last_deep_scrub_stamp":"2024-09-15T21:18:04.752150+0000","last_clean_scrub_stamp":"2024-09-15T21:18:04.752150+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:11:32.218319+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2,3],"acting":[1,0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":6790,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35177,"num_write_kb":101272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4592,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160900,"ondisk_log_size":160900,"up":64,"acting":64,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":26,"seq":111669149879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":708308,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":707771,"kb_avail":93663532,"statfs":{"total":96636764160,"available":95911456768,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4400,"internal_metadata":724758224},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":695316,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":93676524,"statfs":{"total":96636764160,"available":95924760576,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":29,"seq":124554051755,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":613844,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":613307,"kb_avail":93757996,"statfs":{"total":96636764160,"available":96008187904,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4401,"internal_metadata":628027087},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":23,"seq":98784247993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":713812,"kb_used_data":516,"kb_used_omap":4,"kb_used_meta":713275,"kb_avail":93658028,"statfs":{"total":96636764160,"available":95905820672,"internally_reserved":0,"allocated":528384,"data_stored":233930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402,"internal_metadata":730394318},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T21:18:19.677 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T21:18:19.979 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T21:18:19.980 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T21:18:19.980 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T21:18:19.980 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:22.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-15T21:18:26.082 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T21:18:26.082 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T21:18:26.083 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:26.180 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T21:18:26.180 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T21:18:26.180 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:28.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-15T21:18:28.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-15T21:18:32.282 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T21:18:32.283 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T21:18:32.283 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:33.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-15T21:18:33.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-15T21:18:38.384 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T21:18:38.384 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T21:18:38.385 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T21:18:38.385 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:38.415 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T21:18:38.415 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T21:18:38.416 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T21:18:38.416 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:38.468 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T21:18:38.469 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T21:18:38.469 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:38.522 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T21:18:38.523 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T21:18:38.523 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T21:18:38.575 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T21:18:38.576 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T21:18:38.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-09-15T21:18:38.576 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-15T21:18:38.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-15T21:18:38.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-15T21:18:38.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-15T21:18:38.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:18:38.759 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T21:18:39.313 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:18:39.313 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T21:18:39.408 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:18:39.408 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T21:18:39.486 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:18:39.486 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T21:18:39.577 INFO:tasks.ceph:Archiving mon data... 2024-09-15T21:18:39.579 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/data/mon.a.tgz 2024-09-15T21:18:39.579 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T21:18:39.594 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.6Ij4CRSCjl 2024-09-15T21:18:39.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.6Ij4CRSCjl 2024-09-15T21:18:39.725 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.6Ij4CRSCjl 2024-09-15T21:18:39.837 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.6Ij4CRSCjl is 317KB 2024-09-15T21:18:39.905 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.6Ij4CRSCjl 2024-09-15T21:18:39.920 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/data/mon.b.tgz 2024-09-15T21:18:39.920 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T21:18:39.975 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.NjOo5k2OlC 2024-09-15T21:18:39.975 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NjOo5k2OlC 2024-09-15T21:18:40.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.NjOo5k2OlC 2024-09-15T21:18:40.178 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.NjOo5k2OlC is 317KB 2024-09-15T21:18:40.274 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.NjOo5k2OlC 2024-09-15T21:18:40.289 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/data/mon.c.tgz 2024-09-15T21:18:40.289 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-15T21:18:40.343 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.xkS71MEOI0 2024-09-15T21:18:40.344 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xkS71MEOI0 2024-09-15T21:18:40.461 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.xkS71MEOI0 2024-09-15T21:18:40.544 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.xkS71MEOI0 is 312KB 2024-09-15T21:18:40.639 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.xkS71MEOI0 2024-09-15T21:18:40.655 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T21:18:40.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T21:18:40.782 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T21:18:40.782 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T21:18:40.783 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/remote/smithi181/crash 2024-09-15T21:18:40.784 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T21:18:40.846 INFO:tasks.ceph:Compressing logs... 2024-09-15T21:18:40.846 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T21:18:41.222 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T21:18:41.222 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70451.log 2024-09-15T21:18:41.222 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57534.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.70451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54846.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70451.log.gz 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70997.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57534.log.gz 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69485.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54846.log.gz 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70577.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70997.log.gz 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73980.log 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69485.log.gz 2024-09-15T21:18:41.223 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70577.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50016.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73980.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56568.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49134.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50016.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65408.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56568.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56736.log 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49134.log.gz 2024-09-15T21:18:41.224 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37460.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65408.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59554.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56736.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56610.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37460.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57618.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59554.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52788.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56610.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57114.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57618.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65534.log 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52788.log.gz 2024-09-15T21:18:41.225 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37461.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57114.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59131.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65534.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41973.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37461.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66206.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59131.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68980.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41973.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56232.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66206.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61334.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68980.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51318.log 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56232.log.gz 2024-09-15T21:18:41.226 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72468.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61334.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55812.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51318.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62342.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72468.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45421.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55812.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73938.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62342.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45421.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65324.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73938.log.gz 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71291.log 2024-09-15T21:18:41.227 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65702.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65324.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50226.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71291.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55056.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65702.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57072.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50226.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73182.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55056.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50646.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57072.log.gz 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51990.log 2024-09-15T21:18:41.228 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73182.log.gz 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39701.log 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50646.log.gz 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57660.log 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51990.log.gz 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40308.log 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39701.log.gz 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52242.log 2024-09-15T21:18:41.229 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57660.log.gz 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70325.log 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40308.log.gz 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63980.log 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52242.log.gz 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50940.log 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70325.log.gz 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62468.log 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63980.log.gz 2024-09-15T21:18:41.230 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54762.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50940.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62468.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48294.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54762.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53040.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58038.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48294.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64274.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53040.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63476.log 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58038.log.gz 2024-09-15T21:18:41.231 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45509.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64274.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38743.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63476.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64106.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45509.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40271.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38743.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56484.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64106.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69611.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40271.log.gz 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-09-15T21:18:41.232 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56484.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71880.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69611.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63014.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50898.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71880.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63014.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53124.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50898.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59945.log 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-09-15T21:18:41.233 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44375.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53124.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56064.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59945.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52074.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44375.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51528.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56064.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44039.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52074.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51612.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51528.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70199.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44039.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69905.log 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51612.log.gz 2024-09-15T21:18:41.234 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70199.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55350.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39442.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69905.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37458.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55350.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73896.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39442.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37458.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37458.log.gz --verbose 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.46603.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54720.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73896.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60029.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46603.log.gz 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70283.log 2024-09-15T21:18:41.235 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54720.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73854.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60029.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70283.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63644.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73854.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73854.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69317.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63644.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49512.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73098.log 2024-09-15T21:18:41.236 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69317.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52452.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49512.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73098.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55434.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52452.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67592.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56904.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55434.log.gz 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38243.log 2024-09-15T21:18:41.237 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67592.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70871.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56904.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60494.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38243.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64358.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70871.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54384.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60494.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43766.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64358.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54384.log.gz 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73014.log 2024-09-15T21:18:41.238 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43766.log.gz 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73434.log 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73014.log.gz 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70829.log 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73434.log.gz 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69401.log 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.70829.log: /var/log/ceph/ceph-osd.1.log 2024-09-15T21:18:41.239 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70829.log.gz 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50352.log 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69401.log.gz 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51444.log 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37629.log 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50352.log.gz 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64148.log 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51444.log.gz 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50772.log 2024-09-15T21:18:41.240 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37629.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50562.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64148.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50772.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70703.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50562.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74274.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70703.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.70703.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33408.log 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74274.log.gz 2024-09-15T21:18:41.241 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61376.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73350.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33408.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61376.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.61376.log.gz --verbose 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.50856.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71417.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73350.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69275.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50856.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48756.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71417.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58374.log 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69275.log.gz 2024-09-15T21:18:41.242 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56862.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48756.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46677.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58374.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49680.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56862.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60788.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46677.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67508.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49680.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57786.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60788.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66962.log 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67508.log.gz 2024-09-15T21:18:41.243 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71039.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57786.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44459.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66962.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71039.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62510.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44459.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49008.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56106.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62510.log.gz 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62762.log 2024-09-15T21:18:41.244 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49008.log.gz 2024-09-15T21:18:41.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68434.log 2024-09-15T21:18:41.245 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56106.log.gz 2024-09-15T21:18:41.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73056.log 2024-09-15T21:18:41.245 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62762.log.gz 2024-09-15T21:18:41.245 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69947.log 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68434.log.gz 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44207.log 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73056.log.gz 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69947.log.gz 2024-09-15T21:18:41.246 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49218.log 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44207.log.gz 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69149.log 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67130.log 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49218.log.gz 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33701.log 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69149.log.gz 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62930.log 2024-09-15T21:18:41.247 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67130.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33368.log 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33701.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67382.log 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62930.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60998.log 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33368.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67382.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63602.log 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55140.log 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60998.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63602.log.gz 2024-09-15T21:18:41.248 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62300.log 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60284.log 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55140.log.gz 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61670.log 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62300.log.gz 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.60284.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.71333.log 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60284.log.gz 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66878.log 2024-09-15T21:18:41.249 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61670.log.gz 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72048.log 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71333.log.gz 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58206.log 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66878.log.gz 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69779.log 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72048.log.gz 2024-09-15T21:18:41.250 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57912.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58206.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68896.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69779.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57912.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.57912.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.72006.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48336.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68896.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58501.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72006.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60704.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48336.log.gz 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58501.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.58501.log.gz /var/log/ceph/ceph-client.admin.52914.log 2024-09-15T21:18:41.251 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60662.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60704.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71964.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52914.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53586.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60662.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67256.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71964.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57870.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53586.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54510.log 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67256.log.gz 2024-09-15T21:18:41.252 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57870.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57870.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72762.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52704.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54510.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72762.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46529.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52704.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38115.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68138.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46529.log.gz 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38701.log 2024-09-15T21:18:41.253 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38115.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64526.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68138.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37628.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38701.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66122.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64526.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37628.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58711.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66122.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40123.log 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-09-15T21:18:41.254 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67886.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58711.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74232.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40123.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54636.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67886.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58459.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74232.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66458.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54636.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59849.log 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58459.log.gz 2024-09-15T21:18:41.255 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48462.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66458.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46455.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59849.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48462.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50436.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46455.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39146.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60158.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50436.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51822.log 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39146.log.gz 2024-09-15T21:18:41.256 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60158.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50730.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51822.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51738.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70745.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50730.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54468.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51738.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49428.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70745.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48588.log 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54468.log.gz 2024-09-15T21:18:41.257 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49428.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49428.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48966.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63308.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48588.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68096.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48966.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64190.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63308.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45863.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68096.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52368.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64190.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71796.log 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45863.log.gz 2024-09-15T21:18:41.258 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64694.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52368.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55476.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71796.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47790.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64694.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60746.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55476.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59302.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47790.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45546.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60746.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65828.log 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59302.log.gz 2024-09-15T21:18:41.259 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60830.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45546.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51696.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65828.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60116.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60830.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72342.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51696.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62846.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60116.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44837.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72342.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56694.log 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62846.log.gz 2024-09-15T21:18:41.260 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73812.log 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44837.log.gz 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71922.log 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56694.log.gz 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43833.log 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73812.log.gz 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39257.log 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71922.log.gz 2024-09-15T21:18:41.261 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50478.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43833.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56274.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39257.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40012.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50478.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66500.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56274.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69359.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40012.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64316.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66500.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37043.log 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69359.log.gz 2024-09-15T21:18:41.262 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47832.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64316.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48924.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37043.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73560.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47832.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55938.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48924.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55644.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73560.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58921.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55938.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53334.log 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55644.log.gz 2024-09-15T21:18:41.263 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46381.log 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58921.log.gz 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73686.log 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53334.log.gz 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49386.log 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46381.log.gz 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37317.log 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73686.log.gz 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49386.log.gz 2024-09-15T21:18:41.264 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58080.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37317.log.gz 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43284.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz /var/log/ceph/ceph-client.admin.57702.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61040.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58080.log.gz 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45173.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57702.log.gz 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61040.log.gz 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62132.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45173.log.gz 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72090.log 2024-09-15T21:18:41.265 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51066.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62132.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72090.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51066.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43620.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40678.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz -5 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.68392.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63812.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43620.log.gz 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68392.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.63056.log 2024-09-15T21:18:41.266 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.68392.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54342.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63812.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49722.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63056.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68054.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54342.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57030.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49722.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68054.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.68054.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.39738.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49554.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57030.log.gz 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-09-15T21:18:41.267 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59089.log 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49554.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55392.log 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71459.log 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59089.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68222.log 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55392.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71459.log.gz 2024-09-15T21:18:41.268 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68560.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68222.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51234.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55014.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68560.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61124.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51234.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59215.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55014.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64022.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61124.log.gz 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47958.log 2024-09-15T21:18:41.269 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59215.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56820.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64022.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61922.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47958.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67298.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56820.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69569.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61922.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67298.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58416.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69569.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38636.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-09-15T21:18:41.270 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57240.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58416.log.gz 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48714.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45131.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57240.log.gz 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48714.log.gz 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46492.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45131.log.gz 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72258.log 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2024-09-15T21:18:41.271 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68476.log 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46492.log.gz 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51402.log 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72258.log.gz 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59470.log 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68476.log.gz 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53418.log 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51402.log.gz 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44165.log 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59470.log.gz 2024-09-15T21:18:41.272 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39975.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53418.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46640.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44165.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39975.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46640.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48126.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67004.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74022.log 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48126.log.gz 2024-09-15T21:18:41.273 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49092.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67004.log.gz 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38385.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74022.log.gz 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71375.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49092.log.gz 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72594.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38385.log.gz 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71375.log.gz 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71207.log 2024-09-15T21:18:41.274 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72594.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56358.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71207.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.70031.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71207.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72930.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56358.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65954.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70031.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38850.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72930.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63686.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65954.log.gz 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2024-09-15T21:18:41.275 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38850.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54552.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63686.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65870.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48084.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54552.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54552.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58963.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65870.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65996.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48084.log.gz 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64568.log 2024-09-15T21:18:41.276 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58963.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37799.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65996.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69065.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64568.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38384.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37799.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57408.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69065.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72132.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38384.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65282.log 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57408.log.gz 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62552.log/var/log/ceph/ceph-client.admin.72132.log: 2024-09-15T21:18:41.277 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72132.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50142.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65282.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60872.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62552.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37589.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50142.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43158.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60872.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58669.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37589.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62090.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43158.log.gz 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38073.log 2024-09-15T21:18:41.278 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58669.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53838.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62090.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56778.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38073.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66332.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53838.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54132.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56778.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66332.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33448.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54132.log.gz 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2024-09-15T21:18:41.279 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61628.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33448.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65618.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68686.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61628.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73728.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65618.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48546.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68686.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46048.log 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73728.log.gz 2024-09-15T21:18:41.280 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62804.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48546.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37359.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46048.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62804.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72888.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37359.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38998.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67634.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72888.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65198.log 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38998.log.gz 2024-09-15T21:18:41.281 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51570.log 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67634.log.gz 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53754.log 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65198.log.gz 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66668.log 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51570.log.gz 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50268.log 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53754.log.gz 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49302.log 2024-09-15T21:18:41.282 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66668.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52200.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50268.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72678.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49302.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49302.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68518.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52200.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50688.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72678.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.68518.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60200.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68518.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43033.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50688.log.gz 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2024-09-15T21:18:41.283 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60200.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55182.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43033.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56148.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55182.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56316.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69821.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56148.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56316.log.gz 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55560.log 2024-09-15T21:18:41.284 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69821.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63896.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67928.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55560.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60914.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63896.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47916.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67928.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60914.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.60914.log.gz --verbose 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.49932.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68854.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47916.log.gz 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43159.log 2024-09-15T21:18:41.285 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55098.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68854.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59512.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43159.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50520.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55098.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63434.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59512.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62720.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50520.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56946.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63434.log.gz 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-09-15T21:18:41.286 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62720.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44963.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56946.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64778.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44963.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45419.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64778.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43494.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-09-15T21:18:41.287 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45419.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60578.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43494.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53166.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38552.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60578.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64400.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53166.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72972.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38552.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.64400.log: -- /var/log/ceph/ceph-client.admin.69233.log 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64400.log.gz 2024-09-15T21:18:41.288 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72636.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72972.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66542.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69233.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48672.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72636.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66542.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-mon.c.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66542.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61880.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48672.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69022.log 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61880.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.51192.log -- replaced with /var/log/ceph/ceph-client.admin.61880.log.gz 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.289 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69022.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51192.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51192.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67550.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40234.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43621.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67550.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40234.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40234.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.61250.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56988.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43621.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64820.log 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61250.log.gz 2024-09-15T21:18:41.290 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38469.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56988.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38961.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64820.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38469.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46751.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38961.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59639.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69989.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46751.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55728.log 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59639.log.gz 2024-09-15T21:18:41.291 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74190.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69989.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53502.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55728.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50100.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74190.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57450.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53502.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73140.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50100.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57450.log.gz 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61082.log 2024-09-15T21:18:41.292 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46011.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73140.log.gz 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70913.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61082.log.gz 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61796.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46011.log.gz 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62678.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70913.log.gz 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59723.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61796.log.gz 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61292.log 2024-09-15T21:18:41.293 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62678.log.gz 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40493.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59723.log.gz 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61292.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61292.log.gz /var/log/ceph/ceph-client.admin.73224.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68306.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40493.log.gz 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73644.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73224.log.gz 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71712.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68306.log.gz 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56526.log 2024-09-15T21:18:41.294 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73644.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58290.log 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71712.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56526.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48378.log 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59765.log 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58290.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60956.log 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48378.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr: 96.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59765.log.gz 2024-09-15T21:18:41.295 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61166.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60956.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66374.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73770.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61166.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60326.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66374.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64904.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73770.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39109.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60326.log.gz 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52830.log 2024-09-15T21:18:41.296 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64904.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63938.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39109.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61544.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52830.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51948.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63938.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49470.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61544.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64988.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51948.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48630.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49470.log.gz 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-09-15T21:18:41.297 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64988.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65366.log 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48630.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74358.log 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72804.log 0.0% 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65366.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70073.log 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74358.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66626.log 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72804.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55896.log 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70073.log.gz 2024-09-15T21:18:41.298 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53670.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66626.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55896.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38309.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53670.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46788.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59428.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38309.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69737.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46788.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59386.log 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59428.log.gz 2024-09-15T21:18:41.299 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69737.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38887.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59891.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59386.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49974.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38887.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68264.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59891.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71585.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49974.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74148.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68264.log.gz 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43707.log 2024-09-15T21:18:41.300 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71585.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58332.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74148.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70619.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43707.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45215.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58332.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55224.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70619.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71754.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45215.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.55224.log: /var/log/ceph/ceph-client.admin.39812.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55224.log.gz 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53460.log 2024-09-15T21:18:41.301 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71754.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59681.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39812.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53460.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59681.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59681.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60620.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65240.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44417.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60620.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45423.log 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65240.log.gz 2024-09-15T21:18:41.302 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44417.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44417.log.gz 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43245.log 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45423.log.gz 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67088.log 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73476.log 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43245.log.gz 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63098.log 2024-09-15T21:18:41.303 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67088.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.73476.log: -- /var/log/ceph/ceph-client.admin.72300.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73476.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60536.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63098.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51150.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72300.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51906.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60536.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55686.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51150.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51906.log.gz 2024-09-15T21:18:41.304 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59807.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55686.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52410.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51654.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59807.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51780.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52410.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45005.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51654.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51780.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51780.log.gz 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61502.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70493.log 2024-09-15T21:18:41.305 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45005.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39220.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61502.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64862.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70493.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66794.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39220.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63560.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64862.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57492.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66794.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59005.log 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63560.log.gz 2024-09-15T21:18:41.306 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50604.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57492.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59005.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64736.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48882.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64736.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59259.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56652.log 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48882.log.gz 2024-09-15T21:18:41.307 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58585.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59259.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62216.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56652.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53376.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58585.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38157.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62216.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44333.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53376.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71669.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38157.log.gz 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66416.log 2024-09-15T21:18:41.308 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44333.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45299.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71669.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48798.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66416.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39868.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45299.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48798.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.48798.log.gz --verbose 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.53880.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43493.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39868.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53880.log.gz 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60410.log 2024-09-15T21:18:41.309 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65450.log 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43493.log.gz 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44081.log 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60410.log.gz 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72552.log 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65450.log.gz 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66290.log 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44081.log.gz 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73266.log 2024-09-15T21:18:41.310 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72552.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44123.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66290.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67046.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73266.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69527.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44123.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63518.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67046.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57366.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69527.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63266.log 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63518.log.gz 2024-09-15T21:18:41.311 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57366.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57366.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63266.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62972.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72426.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65660.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62972.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65576.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72426.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59596.log 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65660.log.gz 2024-09-15T21:18:41.312 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60452.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65576.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59596.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67340.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60452.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72384.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39775.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67340.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72384.log.gz 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2024-09-15T21:18:41.313 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39775.log.gz 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37737.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz --verbose 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.67718.log 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68012.log 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74400.log 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67718.log.gz 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65156.log 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68012.log.gz 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57198.log 2024-09-15T21:18:41.314 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74400.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64610.log 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65156.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71543.log 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57198.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53544.log 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64610.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71543.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53544.log.gz 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52662.log 2024-09-15T21:18:41.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43119.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52662.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73392.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43119.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53208.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.53712.log -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51276.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73392.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53712.log.gz 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38468.log 2024-09-15T21:18:41.316 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51276.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66248.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66080.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38468.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73308.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66248.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65072.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66080.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68180.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73308.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65072.log.gz 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74064.log 2024-09-15T21:18:41.317 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68180.log.gz 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.41035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37989.log 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74064.log.gz 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37275.log 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49806.log 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37989.log.gz 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45635.log 2024-09-15T21:18:41.318 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37275.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68812.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49806.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43767.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45635.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53628.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68812.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33488.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43767.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45974.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53628.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37459.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33488.log.gz 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63728.log 2024-09-15T21:18:41.319 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45974.log.gz 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37459.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37459.log.gz -5 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51864.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63728.log.gz 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51864.log.gz 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43959.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40937.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39368.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43959.log.gz 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50394.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71501.log 2024-09-15T21:18:41.320 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39368.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57576.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50394.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48210.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71501.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60071.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57576.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48210.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71165.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43892.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60071.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71165.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71165.log.gz 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58837.log 2024-09-15T21:18:41.321 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43892.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60368.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70661.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58837.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60368.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.60368.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50058.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55308.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70661.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50058.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45341.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70955.log 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55308.log.gz 2024-09-15T21:18:41.322 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45341.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72720.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70955.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38031.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37905.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72720.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38031.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69653.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37905.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-09-15T21:18:41.323 INFO:teuthology.orchestra.run.smithi181.stderr: 95.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38637.log 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69653.log.gz 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66164.log 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65492.log 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38637.log.gz 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74106.log 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66164.log.gz 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73518.log 2024-09-15T21:18:41.324 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65492.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39035.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74106.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72216.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73518.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64064.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39035.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72216.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72216.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54258.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64064.log.gz 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54804.log 2024-09-15T21:18:41.325 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67676.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54258.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46233.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54804.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46418.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67676.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46233.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67214.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48420.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67214.log.gz 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69443.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56190.log 2024-09-15T21:18:41.326 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48420.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72510.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69443.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71838.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56190.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68350.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72510.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66038.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71838.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37233.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68350.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61964.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66038.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66038.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61208.log 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37233.log.gz 2024-09-15T21:18:41.327 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65786.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61964.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5/var/log/ceph/ceph-client.admin.61208.log: --verbose -- /var/log/ceph/ceph-client.admin.61712.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61208.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49050.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65786.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61712.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44249.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49050.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37757.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51024.log 2024-09-15T21:18:41.328 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44249.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37757.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39553.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51024.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52032.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39553.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.69695.log -- replaced with /var/log/ceph/ceph-client.admin.39553.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43032.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52032.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69695.log.gz 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62048.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39405.log 2024-09-15T21:18:41.329 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43032.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62048.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40752.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43031.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39405.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53922.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40752.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44501.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43031.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43031.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56442.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53922.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43495.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44501.log.gz 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52284.log 2024-09-15T21:18:41.330 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56442.log.gz 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43495.log.gz 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55602.log 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52284.log.gz 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71249.log 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55602.log.gz 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55266.log 2024-09-15T21:18:41.331 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71249.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51108.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55266.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.55266.log.gz --verbose -- 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.45089.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46122.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51108.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63140.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45089.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70241.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46122.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67466.log 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63140.log.gz 2024-09-15T21:18:41.332 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62636.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70241.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54300.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67466.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60242.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62636.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63182.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54300.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.60242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45675.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60242.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74440.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63182.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38199.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45675.log.gz 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33606.log 2024-09-15T21:18:41.333 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74440.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67424.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38199.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38924.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33606.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50184.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67424.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58122.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38924.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45826.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50184.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50982.log 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58122.log.gz 2024-09-15T21:18:41.334 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46196.log 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45826.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66710.log 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50982.log.gz 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59173.log 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46196.log.gz 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50310.log 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66710.log.gz 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59173.log.gz 2024-09-15T21:18:41.335 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56400.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50310.log.gz 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68728.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56400.log.gz 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55854.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68728.log.gz 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67844.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39479.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55854.log.gz 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54090.log 2024-09-15T21:18:41.336 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67844.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61460.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39479.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54090.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63224.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61460.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69191.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49638.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63224.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69191.log.gz 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62384.log 2024-09-15T21:18:41.337 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49638.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33231.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44543.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62384.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45257.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33231.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39952.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44543.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45549.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45257.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39952.log.gz 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67760.log 2024-09-15T21:18:41.338 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45549.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70157.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67760.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67760.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52620.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70367.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70157.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52620.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27332.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52620.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69107.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70367.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.tmp-client.admin.27332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65114.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27332.log.gz 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42951.log 2024-09-15T21:18:41.339 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69107.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44627.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65114.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.42951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51486.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42951.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70409.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44627.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49596.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51486.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54972.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70409.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48168.log 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49596.log.gz 2024-09-15T21:18:41.340 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62426.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54972.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57744.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48168.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46344.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62426.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64484.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57744.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52746.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46344.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62258.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64484.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57282.log 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52746.log.gz 2024-09-15T21:18:41.341 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67970.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62258.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54594.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57282.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61754.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67970.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39590.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54594.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62174.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61754.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.39590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64442.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39590.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37947.log 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62174.log.gz 2024-09-15T21:18:41.342 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58627.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64442.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.37947.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.68644.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37947.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68938.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58627.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68644.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53082.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68938.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54930.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52116.log 2024-09-15T21:18:41.343 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53082.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37585.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54930.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52116.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.68602.log -- replaced with /var/log/ceph/ceph-client.admin.52116.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.37585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51360.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37585.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65744.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68602.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.51360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63392.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51360.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54426.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65744.log.gz 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67802.log 2024-09-15T21:18:41.344 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63392.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45937.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54426.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67802.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61418.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52158.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45937.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61418.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61418.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr: /var/log/ceph/ceph-client.admin.66836.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58543.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52158.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53964.log 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66836.log.gz 2024-09-15T21:18:41.345 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61838.log 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58543.log.gz 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64652.log 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53964.log.gz 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48252.log 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61838.log.gz 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52872.log 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64652.log.gz 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71081.log 2024-09-15T21:18:41.346 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48252.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.52872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52872.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53292.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71081.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45803.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53292.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.50814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55770.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55518.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45803.log.gz 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65030.log 2024-09-15T21:18:41.347 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55770.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71627.log 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55518.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 0.0% 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65030.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71627.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.71627.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58753.log 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59047.log 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58753.log.gz 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59047.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59047.log.gz /var/log/ceph/ceph-client.admin.68770.log 2024-09-15T21:18:41.348 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67172.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.68770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71123.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.68770.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.67172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64946.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67172.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.71123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56022.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.71123.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70787.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64946.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.56022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43893.log 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56022.log.gz 2024-09-15T21:18:41.349 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38286.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70787.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73602.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43893.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38286.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.59344.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38286.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.73602.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.73602.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.59344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45674.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59344.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40530.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.45674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66752.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45674.log.gz 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57996.log 2024-09-15T21:18:41.350 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40530.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.66752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63770.log 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66752.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74316.log 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57996.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55980.log 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63770.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.74316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.74316.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72846.log 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.55980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70115.log 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55980.log.gz 2024-09-15T21:18:41.351 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58164.log 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72846.log.gz 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70115.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.70115.log.gz --verbose 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr: -- /var/log/ceph/ceph-client.admin.61586.log 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58164.log.gz 2024-09-15T21:18:41.352 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.61586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65912.log 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61586.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.65912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43371.log 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65912.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63854.log 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62888.log 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43371.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63854.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63854.log.gz 2024-09-15T21:18:41.353 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46159.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66920.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62888.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38785.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46159.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.66920.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70535.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66920.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.38785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33055.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38785.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.70535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46085.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.70535.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33055.log.gz 2024-09-15T21:18:41.354 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57828.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46085.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72174.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.57828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58795.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57828.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.72174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40604.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.72174.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58879.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58795.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.40604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40604.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58248.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58879.log.gz 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62006.log 2024-09-15T21:18:41.355 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.58248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54678.log 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58248.log.gz 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46714.log 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62006.log.gz 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69863.log 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54678.log.gz 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54048.log 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46714.log.gz 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.69863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46566.log 2024-09-15T21:18:41.356 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.69863.log.gz 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.54048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54048.log.gz 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62594.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.46566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46566.log.gz 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43413.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.62594.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62594.log.gz -5 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64232.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.43413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48042.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43413.log.gz 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.64232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64232.log.gz 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63350.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.48042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53250.log 2024-09-15T21:18:41.357 INFO:teuthology.orchestra.run.smithi181.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48042.log.gz 2024-09-15T21:18:41.358 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.63350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63350.log.gz 2024-09-15T21:18:41.358 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.53250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53250.log.gz 2024-09-15T21:18:41.617 INFO:teuthology.orchestra.run.smithi181.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T21:18:42.781 INFO:teuthology.orchestra.run.smithi181.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T21:18:42.974 INFO:teuthology.orchestra.run.smithi181.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T21:18:43.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-15T21:18:43.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-15T21:18:43.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-15T21:18:43.632 INFO:teuthology.orchestra.run.smithi181.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T21:30:59.780 INFO:teuthology.orchestra.run.smithi181.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T21:32:33.216 INFO:teuthology.orchestra.run.smithi181.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T21:32:51.622 INFO:teuthology.orchestra.run.smithi181.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T21:33:46.335 INFO:teuthology.orchestra.run.smithi181.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T21:33:46.337 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-15T21:33:46.337 INFO:teuthology.orchestra.run.smithi181.stderr:real 15m5.436s 2024-09-15T21:33:46.337 INFO:teuthology.orchestra.run.smithi181.stderr:user 6m12.340s 2024-09-15T21:33:46.337 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m17.683s 2024-09-15T21:33:46.340 INFO:tasks.ceph:Archiving logs... 2024-09-15T21:33:46.341 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/remote/smithi181/log 2024-09-15T21:33:46.342 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T21:34:51.883 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T21:34:51.894 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T21:34:51.894 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T21:34:51.959 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T21:34:51.967 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T21:34:51.968 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T21:34:51.994 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +1445us[+1453us] +/- 68ms 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 44 -249us[ -241us] +/- 67ms 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -1335us[-1327us] +/- 73ms 2024-09-15T21:34:52.043 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T21:34:52.043 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T21:34:52.052 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T21:34:52.052 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T21:34:52.061 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T21:34:52.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T21:34:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:34:52.255 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T21:34:52.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T21:34:52.283 INFO:teuthology.task.internal:Duration was 2343.938880 seconds 2024-09-15T21:34:52.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T21:34:52.292 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T21:34:52.292 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T21:34:52.369 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T21:34:52.431 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T21:34:52.431 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-15T21:34:52.431 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T21:34:52.451 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T21:34:52.452 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T21:34:52.531 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T21:34:52.531 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T21:34:52.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T21:34:52.724 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T21:34:52.724 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T21:34:52.762 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T21:34:52.772 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:52.878 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-15T21:34:52.899 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T21:34:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T21:34:52.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T21:34:52.924 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T21:34:52.925 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905657/remote/smithi181 2024-09-15T21:34:52.925 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T21:34:52.990 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T21:34:52.991 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T21:34:53.040 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T21:34:53.050 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T21:34:53.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T21:34:53.059 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T21:34:53.059 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T21:34:53.095 INFO:teuthology.orchestra.run.smithi181.stdout: 265701 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 21:34 /home/ubuntu/cephtest 2024-09-15T21:34:53.096 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T21:34:53.106 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T21:34:53.127 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-lost-unfound mon_election/connectivity msgr-failures/none msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}} duration: 2343.9388802051544 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T21:34:53.128 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T21:34:53.204 INFO:teuthology.run:pass